Douglas Alger-The Art of The Data Center - A Look Inside The World's Most Innovative and Compelling Computing Environments-Prentice Hall (2012)
Douglas Alger-The Art of The Data Center - A Look Inside The World's Most Innovative and Compelling Computing Environments-Prentice Hall (2012)
Douglas Alger-The Art of The Data Center - A Look Inside The World's Most Innovative and Compelling Computing Environments-Prentice Hall (2012)
the Data
Center
This page intentionally left blank
iii
The Art of
the Data
Center
Douglas Alger
Visit www.informit.com/title/9781587142963 to learn how to register this product and gain access to additional content.
viii
Preface
The Art of the Data Center
Acknowledgments
Several Data Center professionals generously shared their time and wisdom for this book. Thank you to Don
Atwood of Intel; Sergi Girona of the Barcelona Supercomputing Center; Jon Karlung of Bahnhof.; John Killey
and Jerry Walsh of Citi; John Manville of Cisco; Shawn Mills of Green House Data; Chris Molloy of IBM;
Phil Nail of Affordable Internet Services Online; Dean Nelson of eBay; Christina Page, Nick Holt, Paul Bonaro
and Bob Lyte of Yahoo!; Marc Parizeau of Calcul Qubec; Jay Park of Facebook; Chris Sedore of Syracuse
University; Mark Skiff of NetApp; George Slessman of IO; Jim Smith of Digital Realty Trust; Ben Stewart of
Terremark; Tom Struve and Lon Andersen of ACT, Inc.
Thank you also to their colleagues and associates who scheduled interviews, obtained images and clarified
infrastructure details for each Data Center profile, patiently fielding my e-mails for just one more thing.
These included Xavier Gonzales of Terremark; Scott Gomer, Nancy Owen and Steve Kayser of ACT;
Sara Ibez Leciena and Renata Gimnez Binder of the Barcelona Supercomputing Center; Alex Kirschner
of The OutCast Agency; Elizabeth Kubycheck, Lourdes Pagan and Lisa Krueger of IO; Mike Lewis and
Monica Penrose of eBay; Bruce McConnel and Todd Traver of IBM; Debbie McIntyre, Tina OMara and
June Szymanski of Syracuse University; Rich Miller and Lauren Williams of Digital Realty Trust; Kristine Raabe
of Intel; Denise Schmitz of Neumann Monson Architects; Janis Tarter of Citigroup; Amy Thon of KJWW
Engineering Consultants; Kristian Thorin of Bahnhof; and Dyani Vanderhorst of Voce Communications.
Essential Details
ORGANIZATION: ACT, Inc.
Location: Iowa City, Iowa
Online: March 2008
Notable features: Closed-loop geothermal cooling, HVAC dry coolers, tornado-resistant structure,
hot and cold aisles. Active-active configuration with a second Data Center 5 miles (8 km) away, linked
by ACTs own dark fiber. LEED-Platinum certified.
Time to design and build: 13 months
Size: 6,700 sq. ft. (622.5 sq. m) total, with 4,000 sq. ft. (371.6 sq. m) of hosting space.
Power: Two .9 MW feeds; IT usable capacity of 270 kW
Tier: III
Cabinet locations: 150
Power Density: 6 kW average per cabinet, 11 kW maximum
Infrastructure Delivery: Structured cabling above cabinets. Electrical conduits and air cooling delivered
under a 36-in. (91.4 cm) deep raised floor.
Structural loading: 300 lb. per sq. ft. (1,464.7 kg per sq. m)
Fire suppression system: Double pre-action wet sprinklers, FM200 gaseous fire suppressant. VESDA
detection system.
Leave it to a testing organization to achieve the highest
Background
The Interview
What role does this Data Center serve for ACT?
Tom: We realized the need to establish a second Data Center to ensure the
uninterrupted technology services to ACT Corporation and clients. This serves
that important role in terms of having the redundancy and resiliency in the
system. Just like every organization, technology is just such a huge, integral,
important component of ACT as it is with other modern organizations that we
just have to have the right infrastructure, resilient infrastructure, and be able to
provide those services on an uninterrupted basis, internally and externally.
5
Lon: An initial thought process was, because of how critical the delivery through
1: ACT
technology was to ACT, ACT and the board (ACTs board of directors) really
recognized we need something heavily for disaster recovery and how we would do
that. So, I think following on with that theme is, as youre seeing the trend in the
industry going away from traditional disaster recovery, the plans that I have right
now are starting to migrate away from Ive got my data someplace now how do
I build other servers and how do I recover to an active-active situation where, to
Toms point, even if one Data Center goes away the other centers may run a little
degradated but the service never goes away at that point in time.
So, from the concept of where they were to where were now conceptually trying
to build across the two centers is how do we get that active-active mode to where
were sharing workload across two different sites.
The fact that you knew that this was going to be part of a Data Center
pair operating in an active-active manner, how did that influence the
design?
Tom: One of the early decision points, as we were putting together our vision
and gameplan for this, was to come to grips with are we going to utilize the
traditional disaster recovery site location? What approach were we going to take in
terms of DR? Went through that process, looked at alternatives, the cost-benefits,
of those traditional approaches. Really as weto cut a long story shortdecided
that the best overall model for us, all things considered, was to establish an active-
active model and we have our Data Centers 5 miles (8 km) apart. Some people in
the DR scene would just shudder at that, Oh, man. It should be 25 miles (40.2
km). It should be in different regions of the country to avoid seismic activity, to
avoid electrical grid issues and on and on. Its all a matter of, as we all know, how
far do you take this?
LEED Certification
Leadership in Energy and Environmental Design (LEED) is a rating system for the design, construction, and
operation of green buildings. Facilities can achieve ratings of certified, silver, gold, or platinum based on
criteria within six categories:sustainable sites, water efficiency, energy and atmosphere, materials and
resources, indoor environmental quality, and innovation and design.
6
within the same city. One of the driving factors was that I was able to negotiate
with the city of Iowa City to allow ACT to install our own fiber between the two
sites. Then that starts getting in to the practical aspects that all start tying together
from an operational standpoint, from a disaster recovery standpoint, by having
those tied together over dark fiber.
Also, obviously, being able to have an active-active situation its not a matter of
having to go to some foreign location, try to bring the thing up, try to bring the
data with you, try to reroute communications to it. Its just seamless. And if you
look at it from a disaster recovery standpoint the onerous task of trying to get
staff to a different location to staff it and so forth, theres that aspect. And then
as we look at it from a day-to-day operation, yeah we could have established it
maybe 25 miles (40.2 km) away from Iowa City. Well that, just on a day-to-day
operations, impacts Lons staff and were not a huge global enterprise where we
have thousands of IT staff. So we boiled all that together and the value of having
that direct fiber connection was just huge from what that allows us to achieve
from an IT perspective and also from a cost perspective. Communication costs for
the bandwidth thats required anymore, even with bandwidth expenses coming
down, is still really significant.
Lon: The primary thing there from a technology standpoint is it opens up all
kinds of doors that arent there when youre beyond this dark fiber limitation. In
fact, when I would go to different executive briefings on future technologies
IBM happens to have one, they typically do it around their storage and they start
talking about replicationsthey always have to back up and stop and rethink
their thought process because theyre not accustomed to companies having that
dark fiber and that bandwidth.
Essentially, to us it all looks like just one Data Center. And the machines could
just as well be sitting right alongside each other on the same floor. So for the
engineers it makes a whole lot easier scenario to start envisioning and working
through as to how do you make this thing work and tie together.
7
1: ACT
Figure 1-2 ACTs Iowa City Data Center features Tier III infrastructure and is linked to another
ACT Data Center 5 miles (8 km) away. Image provided courtesy of Neumann Monson
Architects.
What design parameters did you have in place at the start of the
project?
Tom: As we set our project goals, we decided Tier III however we also said its
going to be tornadoyou want to have it be tornado-proof; the engineers are
quick to point out that FEMA uses the term of near-absolute protection.
In our location, seismic zone is not an issue. Tornado is, from a geographic
location and separation standpoint really probably our largest or one of our
largest risks, separate from electrical zones. So thats why we said we are going to
mitigate that risk by designing it to the near-absolute tornado protection.
8
What tier is the Data Center that this one is paired with?
The Art of the Data Center
Tom: This Data Center is a Tier III and then Tier I on the other side.
I assume that that Tier I site was your primary Data Center, prior to
this Data Center coming online. So, from a redundancy perspective
you took a big step up with the addition of this Data Center.
Tom: Yes. Historically, ACT has gone from outsourcing to pretty quickly being
responsible for a lot of its own data and processing. Across the board all of our
technology infrastructure had a huge project just to upgrade everything and this
was one element of that.
It seems you did that in a big way. Youve got the first LEED-Platinum
certified Data Center in the United States and the first LEED-Platinum
building in the state of Iowa. You obviously werent following the pack
when designing the room. You went from outsourced and Tier I to Tier
III, LEED-Platinum. What made you decide to build the Data Center to
those standards?
Tom: Yeah. The Tier III was, to me, a no-brainer. I guess thats easy to say, but
given our overall vision and where we were and where we saw our future going we
just had to have this resilient infrastructure and that was part of it.
The LEED aspect of it (came about) as we were doing initial almost like pre-
schematic design stuff. Our process is, we have a board of directors. We needed to
secure board of director approval for a project of this nature so I had to assemble
just a basic outline, enough to get a budget number put together and secure board
approval before we would really get serious about the project. Put together a team,
we decided we wanted to pursue the design/build of the project, adopt that model
if the project was approved.
9
1: ACT
a LEED background. We talked about it. Frankly, at the initial stages we said,
Boy, Data Center and LEED, its almost kind of a non-sequitor. That doesnt
necessarily follow. We did recognize and thought we would want to follow that
kind of a design framework but at that point we didnt establish a high-level
LEED goal.
It went to the board and we have a very progressive board and they said Yeah
LEED, yeah! And if were going to bother to do this we think we ought to set a
lofty goal and go for a high level LEED certification on this. It was great to get
the board approval so that we could launch into this thing. On the other hand,
we sat down with the team and said And... theyve set this goal for us.
You talk about one of the challenges of the project. First it was just getting over
the shock of the board setting what one could view as an insurmountable goal.
Back at the time the board approved this, we could discover four LEED-Silver
certified Data Centers but most of them were actually contained within a larger
building, which frankly allows you to achieve a LEED certification easier than, at
least it was thought at the time, a Data Center project.
That really got us started. We quickly got over that and it became a huge
motivation. Everyone on the team, fortunately, was like, Tell us it cant be done
and thats a challenge. It just took off from there.
What are some of the features of the facility that helped you achieve
The Art of the Data Center
1: ACT
is to provide daylighting into staff spaces. Here we were with a Data Center
project that needed to be tornado resistant, all the usual security concerns and
all those kind of things. So that was an interesting challenge. What we ended up
doingagain, when you walk up you dont really notice itso theres a curtain
wall on the exterior of the building, however we established a separate interior
vessel that was actually the tornado resistant portion of the building.
While we incorporated tornado resistant glass, you cannot have at least at this
point (in time)Im sure theres probably bullet-resistant stuff, but we were on
a budget. So we came up with that design, not really a compromise it was just a
design solution.
Thats one where I would say if you were just building a non-LEED building it
just would have been a concrete exterior wall. We put some additional funding,
not significant, in to the project for that curtain wall. However, it really adds
value. I mean, wow, when you walk into the building and the staff lab space and
conference area its nice to have the windows and to get light and be able to look
out. Those are elements that are part of that.
And then from the site and so forth we have restored natural prairie vegetation,
non-irrigable, that kind of thing. Again, its just there. Do you think it looks odd
and its something really special? Not really.
Its just details, details, details and not losing sight of the overall vision. And for us
it was that dual goal.
12
The Art of the Data Center
Figure 1-3 Tornado-resistant glass allows daylight into parts of the building regularly occupied by
people. Image provided courtesy of Neumann Monson Architects.
Just like many of the other things, once you see the answer it just seems so simple.
1: ACT
A lot of times its an elegant solution and simple that is the best. And it works.
Some Data Center operators prefer to keep their Data Centers quite
cool. Others using air economization technology employ warmer
settings to maximize its benefits. With your use of geothermal cooling
and dry coolers, what temperatures do you maintain your Data Center
hardware at?
Tom: Were at 72 degrees Fahrenheit (22.2 Celsius) and 50 percent humidity.
Were not doing that (exploring warmer temperatures) at present. There are some
additional refinements and measures like that that we recognize that we could
easily put in place, we just werent at the point where we needed that. Well spot
check what temperatures are and so forth but we dont have integral rack sensors
and those kinds of things.
You obviously wanted this Data Center to score well on the LEED
scale and the system can be specific about what design elements it
awards points for. While LEED has been embraced by the Data Center
industry more than any other environmental building assessment
system, there is some hesitation that features it awards points for
such as daylighting and providing shower facilities and bicycle storage
lockers dont reflect how green a Data Center truly is. Was there any
point at which you felt like you had to make a decision between
earning more LEED points versus good Data Center design?
Tom: From the standpoint of there being tradeoffs or compromising the Data
Center or the Tier III, no. We did not. Certainly we were faced with a large
number of options and decisions where we had to keep looking at whats the
balance here. We wanted to stay within the green framework but couldnt lose
sight of the Tier III.
14
talked about that, could have done it. As we looked at what it would really take
to achieve that and to really have a good ongoing green roof and so forth we felt
there were some potential compromises with roof penetrations and just some
different structure things that way. We looked at the options and really we were
able to still be within the spirit and framework of LEED by instead putting
on an Energy Star roof system. It has a double roofing system: as far as the
tornado resistant structure we have a conventional roof, a couple inches (about 5
centimeters) of concrete slab over the top of that and then a single-ply membrane
roof on top of that, which is the Energy Star rating.
As you say, yes we provided the shower facilities, the various other kind of things.
It didnt compromise the functionality of the building.
Its a good question, Doug. Its right on target. As you know, LEED is constantly
evolving and refining and I know there have been efforts to establish a designation
for Data Centers. Personally I think that makes a lot of sense.
Certainly that was a challenge taking the LEED framework as it existed at that
point in time and applying it to a Data Center. I would imagine that as the
framework gets developed itll be better. I dont know if you would say easier in
the sense of using the framework that we had. Its just amazing what just a few
years does.
At the time we were doing this, in terms of the materials that were available, that
were certified appropriately, all the different sources for construction materials
we have recycled content in the steel and recycled content in the concrete and
those kind of thingsit was more difficult then to come up with those things
than I think even it is just today not that many years later. I guess the point is, on
the one hand I think the framework will be evolving and becoming more specific,
which I think definitely it needs to be. At the same time, then, I would imagine
that would then get more stringent from a Data Center standpoint. On the other
hand, then, theres a lot of materials and technology that are now coming on-
stream that one could get to make use of.
Among your re-use items you mention the floor system. Did you have
that in another facility or did you go shopping for a raised floor system
from someone else?
Tom: Yeah, we went shopping.
15
Where do you find a gently-used raised floor system for a Data Center?
1: ACT
Tom: At the time there was still a lot of stuff on the open market from the dot-
com bust. And there are sources for used furniture, used raised floors, those kind
of things.
What we did do was to specify and apply new surface material to it. So from an
appearance standpoint, the tile material is new. It looks new and so forth. But the
bulk of it is not the little surface its the actual structural portion of the floor tile.
Figure 1-4 Structured cabling is routed overhead. Electrical conduits and cooling are provided
below a raised floor. Image provided courtesy of Neumann Monson Architects.
16
You routed your structured cabling overhead and your power and
The Art of the Data Center
that through automated controls and sensors or if were aware that were in to a
1: ACT
warning area we can also remotely or at the Data Center put it in that mode.
And then we put the generator inside the building. The sole reason was to protect
it in the event of a tornado type event. Its nice from a vandalism point of view,
too, but its really the tornado thing that drove that. Again, the building could
be standing but most likely in a tornado event power to the building you have
to assume would be interrupted and so we need that generator to be functional.
So its within the building. Then you have the obvious problems of trying to
get a massive amount of air into that generator space for cooling primarily and
exhausting it. And so, on the air intakes and exhaust theres structural (barrier)
the best way to describe it is an array of steel tubes that would prevent a tornado-
driven 2-by-4 or whatever to be able to penetrate in and damage the generator.
Figure 1-5 ACT placed the Data Centers standby generators inside the building to safeguard them
from tornadoes. Image provided courtesy of Neumann Monson Architects.
18
What were the biggest challenges or surprises that came up during the
The Art of the Data Center
project?
Tom: Its good to have a little bit of time probably separating from when we went
through it. At the time youre in it you just do it. In some respects there werent
any huge challenges. It was really just breaking it down element by element
and just working through the options, looking at the LEED framework and
identifying options that we could incorporate.
We were pleasantly surprised by the synergies that ended up happening between
the LEED design and achieving the Tier III. For instance, the geothermal is
just a great synergy there. It has the energy efficiency and so forth. And the way
we use that is in our climate the geothermal field is most energy efficient in the
summertime and then we have dry coolers, out in the HVAC courtyard which
as they say provide essentially free cooling. All it requires is just the pump to
be running out there, so we rely on them as our primary heat rejection source
in the wintertime. Well, as part of the Tier III you have the N+1 redundancy
of the CRAC (computer room air conditioner) units within the Data Center,
however we also wanted to have redundancy and protection in terms of the rest
of the system.
Thats one thing that has always bothered me as I walk up to these Data Centers
is you have all these condensing units outside, up on the roof or out in the
courtyard or whatever, and in terms of potential risk from a tornado event or
vandalism or even a car running into the things, that kind of stuff. Thats always I
thought a risk thats not really a lot of times appropriately addressed. Sometimes
theyre placed within the building and then you have a huge challenge, which can
be overcome, of getting sufficient airflow in there for them to operate.
So with the geothermal everything is under the ground and the pumps are
all within the tornado-protected vessel. Worst case we feel that in the event
of a tornado we might lose some of those exterior condensing units but the
geothermal is standalone, self-sufficient and can go. Conversely, if we have some
problem with the geothermal we feel that needs addressingyou get a leak in one
of the loops or somethingif you need to take that offline then no problem we
just go over to the other side. So thats a nice synergy, I think.
Theres the energy-efficiency aspect to both of them, but you put them both
together and from a seasonal energy efficiency standpoint and then from a
resiliency standpoint in terms of the Data Center operation I think its pretty cool
how that works together.
19
Do you think the geothermal system and the other design elements
1: ACT
that helped you achieve LEED-Platinum certification are universal
solutions? Are they things that anyone building a Data Center could do
now and should do now, or are some of them very specific to what you
were trying to accomplish and where your facility is located?
Tom: I think the geothermal and combining that with the dry coolers is
definitely a climate-related solution. If your Data Center is in Texas, it really
wouldnt be a model that would fit. Above a certain latitude, then good to go.
There are some things dependent that way.
Otherwise I would say the rest of it would be just pretty universally applicable,
adaptable.
If you could go back and design this Data Center all over again what,
if anything, would you do differently?
Tom: To date, nothing has come up. Pretty surprising.
Lon: One of the things that hasnt been mentioned is there is already pre-
planning on expansion of this that got engineered and thought about before we
went ahead. Even to that standpoint, if the capacity got beyond where we needed
it we already had that thought into and laid out from an architectural standpoint,
and pre-planning how that next piece of expansion would go. And Im not sure
that facilities typically do that very well.
Tom: Well see how it develops, only time will tell. From my standpoint, our
future roadmap would be to establish a similar Tier III facility on campus to
replace the Tier I Data Center, which is contained within an office building and
constrained.
inside the center, perhaps how we do backups, what we would do those to, and
The Art of the Data Center
how that story would bethats the biggest thing that if I were to look at it I
would say that design or that thought process needs to be ongoing and developed
as the building is being constructed.
Do you have a target level for how much virtualization you employ in
the Data Center?
Lon: Not a particular target level at this time. I can tell you the intent is to
reduce this year (the number of physical servers) by about 50 percent and we will
reduce further as we go on. Were going to reduce down from about 450 physical
servers to in the neighborhood of 200.
That will come down to a point of, like most places, a percentage of utilization
and that saturation of where youre actually providing decent core services versus
how much youre trying to put within a particular device.
Figure 1-6 ACT is reducing the quantity of servers in its Data Center through virtualization. Image
provided courtesy of KJWW Engineering Consultants.
21
Any other Data Center project strategies that you can recommend?
1: ACT
Tom: Establishing clear goals for the project. Its one of the obvious, simple
things but man you better know what they are and keep grounding it toward that.
The other thing that I constantly say is assembling the right team to achieve the
goals. Like we all know, it just comes down to the people involved and working
as a team on a project like this. Understanding what your goals are and then
choosing the right team members that bring the right experience and the right
kind of mindset to the project is just so important.
And I think that being motivated, certainly at least on this project,
being motivated by having an apparently very difficultwe say almost
insurmountabletask. Being open-minded and creative.
And just that details, details, details, while at the same time keeping a clear grasp
of the overall goals and vision. In terms of the (LEED) Platinum, certainly, boy,
it just really gets involved in the details of the thing. Like I said earlier, too,
understanding best practices sufficiently to be comfortable making decisions
regarding which of those to incorporate and which of those to reinvent.
Also a really key thing at least on this project was owner decision making.
Ultimately this is the owners project. You have to be knowledgeable and
intimately involved. If you think you can just hire a bunch of consultants and
say Hey, heres what I want. I want LEED-Platinum and I want whatever tier
and these kind of things, thats just not going to happen. Youre not going to be
satisfied overall with the project and its not going to achieve what you need it
to in all respects if youre not really involved. And you have to be knowledgeable
about the subject matter. You need to make the tough decisions in a very timely
manner and as an owner you need to accept full responsibility for the risks
associated with non-traditional approaches. And in doing so, the owner really sets
the tone by making those decisions.
If you get into a situation where youre setting up for a bunch of finger-pointing
later if something doesnt work then that just causes everyone to be defensive and
to fall back in to that old deal of just using the safe, tried and true best practices.
To me, my bias is its not the place for decisions by an owner committee.
Someone needs to be empowered with the authority and the responsibility to
make those decisions and just to make them.
We would to keep this thing moving and just a whole array with the LEED-
Platinum framework and since we were in somewhat uncharted territory we
would talk about things, bring them up, get them vetted and then I would just
22
make a decision. I know sometimes they would say Dont you want to think
The Art of the Data Center
about that? but weve all investigated, all the information there is to know is
here, so boom lets go this direction. We need to keep this moving. Boy, you can
just get so bogged down in something like this that you just have to be willing to
stand up and make decisions and move on.
What do you think were the toughest decisions for this project, or that
someone else in a similar project might find the toughest to make a
call on?
Tom: I suppose because the project was successful, both from the LEED
standpoint and the daily operational perspectiveits just performing greatin
hindsight they dont seem like there were any hugely difficult ones. Certainly
committing to the geothermal design, which was at that point not really proven.
The others were just evaluating the options, the pros and cons, and understanding
the dual goals and going for it.
A team dynamic that really workedin terms of achieving it, its more the people
involved and setting the toneis setting egos aside. And they did. As we were
working through all these different options and alternatives you could have seen it
stepping on toes of different disciplines. But everyone just really worked together
towards the common vision and were just really open and creative and willing
to sit and listen and consider different ways to do it and then talk about how
maybe mechanical and controls and electrical had to come together on some of
these things. How best to do that as we tried to incorporate the flexible, modular
approach into things.
23
1: ACT
approaches. I think sometimes you just get some of these egos. It can enter in.
You get someone people in and they feel it has to be their way. Thats easier said
than done to get past that.
Figure 1-7 Chilled water piping for ACTs Iowa City Data Center. Image provided courtesy of KJWW
Engineering Consultants.
Figure 2-1 Affordable Internet Services Onlines Data Center operates entirely on solar power.
Images provided courtesy of AISO.
Chapter 2
Affordable Internet Services Online (AISO)
Essential Details
ORGANIZATION: Affordable Internet Services Online (AISO)
Location: Romoland, California
Online: April 2005
Notable features: Solar power used as primary power source. Air conditioners use atmospheric energy
process to produce cold air. Solar tube lighting and a rainwater collection system.
Time to design and build: 6 months
Size: 2,000 sq. ft. (185.8 sq. m) total, with 400 sq. ft. (37.2 sq. m) of hosting space
Power capacity: 110 kW from on-site solar array
Tier: III
Cabinet locations: 15
Infrastructure Delivery: Power, cooling, and cabling infrastructure are provided overhead
Structural loading: Undetermined
Fire suppression system: Aerosol
Solar power is an impractical source of energy for a
Background
The Interview
Figure 2-2
AISOs small
number of servers
makes it possible
to run entirely on
solar energy.
department doesnt want to do it doesnt mean all of the other divisions do, and it
Talk a little bit about the cost issue. I assume you mean the upfront
capital cost to install a solar system. Did you find the system to be
notably more expensive than a conventional power system?
Oh sure, yeah. Im not sure what our cost was, it has been so long. But I can tell
you that it definitely is very expensive to do and there are a lot of things that
need to be fine-tuned after you install it. There are a lot of things that you have to
figure out that you hope will work and theres a lot of trial and error.
Getting started is extremely expensive but in our case it has paid off. Most
companies look at the ROI before they invest and the ROI (return on investment)
in most cases can be 10 to 15 years.
Any potential errors that you would alert someone to if they were
The Art of the Data Center
With that extra cost up front, did you have a sense of your return on
investment and how quickly you would make your money back due to
reduced energy costs?
We didnt even take a look at that. Most companies they want to know what the
ROI is before they even think about doing it. We just did it because we thought
thats the right thing to do. We figured it would pay off in the long run we just
didnt know how long.
Another way that youre leveraged solar power is through the use of
solar tubing and windows. Did you have any security concerns about
using them?
No security concerns at all. The solar tubes are small in diameter so no one can
simply knock off the top and climb into the Data Center.
31
Figure 2-3
Tell me about your air conditioning system. I understand that the units
you have installed actually work more efficiently in warmer conditions.
Our AC units work extremely well 99 percent of the year. The other 1 percent we
need to run regular AC to help knock down the temperature inside. The reason is
the AC units use water and do not work well when the humidity outside is high.
The units we use are water cooled and can take 110 degree (43.3 Celsius) outside
air and cool the air down to 65 degrees (18.3 Celsius).
They work very, very well. Theyve been a really good thing for us. Because they
use water we have some large tanks we use to catch the precipitation when it does
rain out here. We collect that water and pump it to the Coolerado switch and it
in turn cools the Data Center. It only has about two moving parts, so the average
power consumption is only about 200 watts, which is nothing.
32
We used to use a cooling system called Freus, which was water-cooled as well but
The Art of the Data Center
it was down more than it was up. Our air conditioner guy was looking online
one night and he happened to find Coolerado. We went down and saw them at a
trade show and loved what we saw so decided to go for it.
Figure 2-4 AISOs air conditioners use an atmospheric energy process to produce cold air and are
more energy efficient than conventional units.
I was amused to learn that you have tiny wind turbines in the ducting
of your building. Where did you come up with the idea for that?
It just came to us. We said This looks like a pretty good idea. Lets just try it.
A lot of the things that we do, it just comes to us in the middle of the night or
whenever, and we just go for.
33
The power from the turbine is used to charge small batteries (which then power
Speaking of that, please talk about how you are using virtualization
technology.
Virtualization has played a major role in the running of our solar powered Data
Center. First off we had to eliminate all the high energy computers, servers, and
equipment. Then we decided on using NetApp for our SANs as they are the most
reliable and most energy efficient. We chose the IBM servers as they were the
most energy efficient servers we could go to. They are running the most energy
efficient power supplies and the most energy efficient processors on the market.
One hundred percent of our Data Center is virtualized, which also makes it
extremely easy to manage.
Weve gotten away from having physical boxes for every single thing that we do. It
just wasnt efficient. We use virtualization for our local computers and we use that
for all of our clients.
We started off with physical boxes and we migrated to virtual machines. We have
chosen VMware to go in to our virtual infrastructure. We started off with iSCSI
and that was a nightmare, because that didnt work very well. After we had 100
or so servers migrated across, we had a lot of slowdown issues so we had to get
off iSCSI real quick and move into a different protocol. That was really the only
challenge that we had. Other than that it has been just the best thing ever.
Its very, very easy to manage. It has made our life so much better. If a client server
goes down and its 2 oclock in the morning, we can just VPN in, get on and do
a simple reboot. Bring it right back online from our cell phone. Its very easy,
secure, and simple to manage.
Its so reliable. It just runs. I could never go back to physical servers again.
Figure 2-5 Use of virtualization has allowed AISO to expand its computing power in a small
physical footprint.
35
Obviously not all Data Centers are using virtualized servers to the
Figure 2-6 AISO harvests rainwater to cool its Data Center and for landscaping. Collection tanks
can hold 10,000 gallons (37,851.1 liters) of water.
You started with the solar infrastructure and then put in other
infrastructure efficiencies over time. What challenges, if any, did you
have in introducing new technologies? Were there any surprises along
the way?
There werent really any surprises. I think the biggest thing for us was just
explaining to clients that the virtualization was a better technology than having a
physical box. That was the hardest part for us.
It took a little bit of convincing them. Now, people just love it. Best way to go.
37
If you could go back and design your Data Center again what, if
anything, would you do differently?
Ive looked at that quite a bit over the years and I dont think really much of
anything. It just fell in to place very nicely the way that we did it. Other than
maybe make it a little bigger, thats about the only thing. The design worked out
really well. Were very pleased with it.
There are some racks out there that were looking at, kind of testing out. They
The Art of the Data Center
dont stand straight up like your typical racks do, they lay down. You wouldnt
need any cooling whatsoever to use them. You actually take the fans out of the
servers. The racks themselves do the coolingyou close the lid on a rack and
its a self-contained system. Were taking a look at those and trying to see how
they might fit here. Were not sure at this point, but it sounds like a pretty good
concept.
What design advice would you offer to someone looking to build their
own energy-efficient server environment?
I would recommend if they start from scratch figure out how much actual energy
they need and build your alternative energy solutions around those numbers.
Next I would look at how you would setup and handle the cooling and heating
of your Data Center. Next, how will you provide light in your Data Center? How
will you run your servers in the most energy efficient ways and how will you do
this being fully redundant?
Look at the insulation that you build into the facility itself. The more energy
efficient it is, the less your cooling (system) is going to have to work.
And then, try to virtualize as much as you can. The problem with a lot of the
Data Centers is that they dont virtualize so they just use a tremendous amount of
power.
2: Affordable Internet Services Online (AISO)
39
Figure 3-1 The above-ground entrance to Bahnhofs Data Center in Stockholm, Sweden. Embedded
100 ft. (30 m) underground in a former nuclear bunker, the Data Center retains its
wartime codename, Pionen White Mountains. Images provided courtesy of Bahnhof.
Chapter 3
Bahnhof
Essential Details
ORGANIZATION: Bahnhof
Location: Stockholm, Sweden
Online: September 2008
Notable features: Artificial daylight, greenhouses, waterfalls, and a 687 gallon (2,600 liter) saltwater fish
tank. Two Maybach diesel submarine engines used for standby power.
Time to design and build: 20 months
Size: 10,764 sq. ft. (1,000 sq. m) total, with 5,382 sq. ft. (500 sq. m) of hosting space and
2,153 sq. ft. (200 sq. m) for back-of-house systems. Remaining space is for office and personnel areas.
IT-usable capacity: 800 kW
Tier: Not stated
Cabinet locations: 140
Power Density: 5.7 kW average per cabinet, no specific maximum
Infrastructure Delivery: Cooling, structured cabling and electrical are delivered below a 3.3 foot (1
meter) deep raised floor.
Structural loading: 403 lb. per sq. ft. (2 tons per sq. m)
Fire suppression system: Novec 1230
Anyone who sees Bahnhofs Data Center in Stockholm
Background
The Interview
What made you choose a bunker as an ideal location for a
Data Center?
There were several reasons for it, but the primary one I think (is that) we had built
five Data Centers and most of them were built in traditional environmentsold
warehouse redesigns and old office building redesignsand we wanted to do
something different.
43
We are geologically in a very stable area. Most parts of Sweden are built on
3: Bahnhof
granite. The mountain, the stone is solid rock and its 2 billion years old. Its one
of possibly the earths oldest and most stable from a geological perspective.
There are fortresses and stuff from those days of the Cold War and I looked
through most of them. I think I was in 40 different spots. Most of them were too
small or scrapped or used for something else, but this facility was still there. We
were very lucky to get hold of it.
If you work with computers, you realize that the threat to operational computers
is not too much. The physical protection, okay, it must be there of course. It must
be there, but now we have a mountain defined originally to (withstand) well not
a hydrogen bomb outside the door but at least a hit somewhere in Stockholm and
the computers probably survive even if people may not.
The place was very big and it was also located very central in Stockholm. Our
competitors, and many Data Centers, are often located way outside the inner city
urban area because they can be built in a larger complex outside for a cheaper
price. But, you see, many computer consultants they live in the town. They dont
want to travel far away to look at their boxes. Its nice to have them conveniently
close to the city, actually. Of course its very hard to find a great, cheap space
where you can build a Data Center in this urban city but with one exception and
that is these caves because nobody can use them for something else.
Sometimes they have been used as garages, but (at this site) the entrance was too
narrow so it was impossible to build something like that. I would say it was a
clear benefit that the space was so centrally located and that gave us an advantage
against our competitors. Our clients can easily access the computers and after they
have accessed it they are in the main city area.
Its very hard to find a great, cheap space where you can build a
Data Center in this urban city but with one exception and that is
these caves...
44
The Art of the Data Center
Figure 3-2 The Bahnhof Data Centers Network Operations Center (NOC). Artificial daylight,
greenhouses, and even a saltwater aquarium have been installed to avoid the impersonal
feel of many conventional Data Centers.
The second reason: of course, the physical protection was also great. As you
know, when you work with a computer center, it might not be of that huge
importance. Most errors in my work are basically caused by humans. Human
errorthat is the most frequent. Mistakes in configurations and whatever can
make your environment go down. Not from some kind of big trouble with the
physical protection.
But that doesnt matter because the clients they like what they see. From a
marketing perspective, theyll of course appreciate that they feel secure. Even if I
know that humans are more cause of problems than the physical location. They
appreciate the solid feeling of the rock. The solid feeling of the rock gives us an
advantage compared to a conventional center.
45
The third thing was a combination. It was fun to do it. It is like playing. The fun-
3: Bahnhof
ness of it also brought us an advantage in marketing. I mean, I speak to you now
and we have had a lot of magazines and footage from different media channels
covering it. And of course that has given us invaluable access from a marketing
perspective. If I had to spend the amount of money on advertising it would
have been impossible. Now we have all of this, if not for free, it has given us an
advantage from the marketing perspective.
The style we chose, that was about the fun-ness of it. Its a French architect who
has been responsible for the design, but he has been working very close to us and
we have discussed ideas.
The great inspiration has been one-part James Bond and one part Star Trek or
science fiction movies and stuff like that. That has been an inspiration source
to do it like this. The only thing that is missing, I was considering a white cat
with long hair, like the character in James Bond, that Blofeld guy (Ernst Stavro
Blofeld), and sitting in our conference room in glass and have this cat. We have
plants, but no pets right now.
Figure 3-3 A circular, glass-walled conference room overlooks the Bahnhof Data Centers
server rows.
46
This room definitely makes an impression upon people when they see
The Art of the Data Center
it. Did you have a sense when you were designing the Data Center that
it would be so attention-grabbing?
Absolutely. There is something magical about caves. I dont know what it is but
people are sucked in to the facility and immediately like it. Even when we were at
a very early stage.
We defined it completely from the beginning.
We have blown out 4,000 cu. m (141,300 cu. ft.) of stone additionally, on what
it was before, with dynamite. We were building it for almost for 2 years. The
neighbors in the area were not so happy from time to time from this dynamite.
Once accidentally there was an explosion and the coverage from the front door
was not enough so a small piece of stone penetrated a car outside the entrance.
Luckily there were no people there and nobody was harmed.
When we had it very early people were very interested in it and they wanted to
come and see it and looked. Right now we have delegations from companies
and universities and interesting parts. And our policy has always been to be very
open about it. We invite people and we have meetings and visit the facilities
not where we have the servers, in that space, but in the other parts of the facility
where you can see it very well.
How long overall did it take to design and build the Data Center?
Maybe two and a half years from the start. We hadnt built a Data Center in solid
rock before and this was so different. It depends upon where you build it. If its
in a warehouse it takes a certain amount of time but this was so special. It took us
about twice as long as we foresee from the start.
This was a success. It has been a commercial success and its a success from a
design point (of view), from all angles, actually. Even so, Im not sure if I had
another opportunity to build a similar Data Center again I dont know if I would
do it because it was so tiresome to remove all this stone, and it was a risky project.
It could have gone wrong. There could have been problems with water from the
groundwater. There could have been different types of problems. It is cool and it
has been a success both technically and commercially but it was a lot of work to
do it.
47
I remember when I presented it (the proposal) to the bank, when we had some
3: Bahnhof
investment loans to carry some of the initial investment, and I told them that we
were going to build a James Bond fortress in the mountain. This is long after the
dot-com time. It might have sounded like a dot-com project.
Well, it turned out good and we had investments that are not huge. We kept a
very tight budget so it was built with very little money, actually. We put in mostly
our own work and found solutions ourselves to do it.
Figure 3-4
Solid rock
surrounds the
server cabinets
within the Bahnhof
Data Center.
48
Its very hard to build individual coolness in individual cabinets unless you know
3: Bahnhof
exactly what type of computers or configurations people are using. When youre
doing open co-location space it must be very flexible. We have clients even with
old power models, God forbid, but that is the case. And the client is always right.
Then it has shown the most convenient way to do it is to have this level floor
cooling model.
In all other aspects its a conventional Data Center from a technology perspective.
We have great redundancy. The network is coming in from three different places.
The electrical power is also coming in from different places. The unique stuff is
the atmosphere and the protection it gets from the mountain.
At the early stage we discussed the use of some kind of geothermal cooling: you
drill into the mountain and bring up coolness from the ground. This turned out
to be insufficient, we have too much energy. We investigated that but I was told
we would have had to drill under several hundreds of homes and it would have
been of effect for a limited time span. Sooner or later we would eventually warm
up the entire mountain and then (the geothermal cooling) would have no effect.
So we decided to use the outside air cooling instead. One advantage in Sweden
is that the climate is colder, so we can very often use the outdoor temperatures in
the cooling system. We use conventional cooling also but using free cooling from
the outside air.
Figure 3-5
The
underground
Data Center
incorporates
outside air
cooling rather
than geothermal
cooling, to avoid
warming the
mountain that
surrounds it.
50
Did you have any particular challenges around ventilation and using
The Art of the Data Center
Did you know from the beginning that you wanted to have those sort of
features in a facility or did that come later?
Definitely. That was initially planned. They were in the budget from the
beginning. It was a strategic decision very early. We decided that this should be a
nice James Bond fortress and if you have a James Bond fortress you should have a
fish tank.
We originally were planning to have piranhayou know, the little fishes that eat
meat. But I was told that they are so boring. They just stand still and are not so
colorful. So we decided for saltwater fish and colorful instead. And, the plants,
I think it was a science fiction movie Silent Running, something from the early
70s, where we had these space stations with greenhouses floating around in outer
space. This was the original idea. Everything was defined with this in mind. It was
not added on later. It was the idea to have all this from the beginning.
51
3: Bahnhof
Figure 3-6 Artificial waterfalls within the Bahnhof Data Center.
Its impossible to argue with the logic that if youre going to have
a James Bond villain lair you should try to include fish tanks with
piranha in them. Thats certainly consistent. You mentioned the raised
floor system for the delivery of cooling. Is the Data Centers power and
cabling infrastructure delivered under the floor as well?
Yes, that is under the floor. That is why we lowered the floor so much, to give
us additional space. I know that you can have it above, but aesthetically we
didnt want to have too much stuff (visible). It should be clean and look science
fiction-properly. That is why we decided to have them under the floor. It is a
higher (deeper) space than normally so this gives good airflow flowing in. I know
sometimes you get problems with the cables under (a raised floor) because they
can hinder the airflow, but this didnt happen. The raised floor is definitely higher
than in our conventional Data Centers.
52
If you could go back and design the facility all over again, is there
The Art of the Data Center
3: Bahnhof
Figure 3-7 Two Maybach diesel submarine engines, complete with accompanying sound-horn,
provide standby power for the Data Center.
54
Do you have plans for any future Data Center buildoutsfor any other
Bond villain lairs if you will?
Yes. We are working on a concept with modular Data Centers. The idea sprang
from that fact that if you build in a mountain you realize that the mountain
it is very hard to expand in space. Once the space is finished you have to do
something else. So, I would like to build Data Centers in modules where you can
have module after module after module when you need more space.
55
With those modules, are you also going to put in any interesting
3: Bahnhof
features?
Absolutely. Absolutely. Im considering building a space theme. Imagine a space
station on Mars or something like that, with modular tents and very scaled
design. It should be defined.
If you consider the Swedish products like the Absolut vodka bottle. We can
always say that the bottle is a bottle. But if you add some design and add some
concept with it it enhances the experience. Its the same. If I say We are going to
build modules people will believe its containers, but its not containers. They will
be designed differently and they will for sure give the fun factor we are looking
for.
I have gotten a lot of publicity for this project. A mountain is a mountain, so
maybe that is hard to top in terms of the atmosphere. But we will give our best to
do it.
Any final advice that you would offer to others who work on Data
Center design projects in the future?
I think you should give the human factor a great thought. Computer centers
should be built for humans and for their clients. Most often you focus a lot on
the technological aspects but you forget about the humans who are going to work
there. Also, giving this design you add an experience for the client. That is very
often forgotten in these type of projects. Bring a human touch to the computer
center, thats my advice.
56
The Art of the Data Center
Figure 3-9
The reception
area at the
Bahnhof Data
Center.
57
Figure 3-10
3: Bahnhof
Greenery is planted in
the Network Operations
Center (NOC) to make
the environment more
comfortable for employees
to work in.
Figure 3-11
The Bahnhof Data Centers fire suppression system features a gaseous suppressant.
Figure 4-1 The Barcelona Supercomputing Center resides in the Torre Girona Chapel. Images
provided courtesy of Barcelona Supercomputing Center.
Chapter 4
Barcelona Supercomputing Center
Essential Details
Organization: Barcelona Supercomputing Center Centro Nacional de Supercomputacin
Location: Barcelona, Spain
Online: April 2005
Notable features: Server environment is located in 1920s chapel and houses the MareNostrum
supercomputer
Time to design and build: 7 months
Size: Overall chapel floor space undetermined. Hosting area is 1,722 sq. ft. (160 sq. m)
Power: 1.4 MW used by the entire facility, with about 850 kW consumed by computing hardware
Tier: 0
Cabinet locations: 48
Power Density: Varies by cabinet, up to 22 kW maximum per cabinet maximum
Infrastructure Delivery: Electrical conduits, cooling, and cabling all delivered under raised floor
Structural loading: 512 lb. per sq. ft. (2500 kg per sq. m)
Fire suppression system: Water mist system
When Spanish government officials needed to find a
Background
The Interview
What drove the decision to locate your supercomputing center in the
chapel?
In 2004 the Spanish government had the opportunity to buy MareNostrum,
the supercomputer. The reason that Spain was able to do so is because a large
supercomputing research group from the Technical University of Catalonia was
working, since 1980, with very good cooperation with IBM. This offered the
Spanish government the chance to get them the system. But then the deal was to
have the system installed in four months very close to this research group, that
is, in the Technical University. So, as you can understand, a new computer room
setup in four months requires an existing building, but an existing building with
some special facilities. At this point of time there were only very limited spaces
available and one of them was the chapel, because it was a clear floor plan of more
61
than 120 square meters (1,292 square feet) with no columns; a height of more
The Top500 ranks the fastest supercomputers in the world using what is known as the Linpack Benchmark, a
Note
method of measuring a computers floating point rate of execution by running a program that solves a series of
linear equations. The Top500 is updated twice per year, at the International Supercomputing Conference (ISC) in
Europe each June and at the Supercomputing Conference (SC) in the United States each November.
So in reality, we started the design of the system in April 2004 until June, with
construction spanning from July to October. This construction include the
computer room itself, the site facilities with the control switches and fire detectors
and alarms, and also the electrical transformers, because we are connected to a
medium voltage ring so we have to transform this to normal voltage.
62
Even with that outside delay thats still a relatively short amount of
The Art of the Data Center
Figure 4-2 The supercomputing center resides in a large glass box, providing maximum visibility of
the surrounding chapel.
Speaking of the glass box, why did you enclose the Data Center in
that way?
With the building completely finished the architect told us that it is simply
furniture inside the house, inside the chapel, because we dont want to modify
anything on the chapel itself. If you are installing some walls inside the chapel,
you will destroy the image completely. If it is glass, on the other hand, everything
is completely visible on the other side, so the chapel is always there.
65
But there is also another reason. In Spain we installed the computers, called
Your power, cooling and cabling infrastructure are all delivered under a
raised floor, yes?
Yes. Every one of those racks is using 22 kW. At that point of time the cooling
was by air, so we decided to have a 97 cm. (38.2 in.) elevation for the first floor
and all the facilities and all the services went on this first floor.
"Having walls that hide the chapel would not be a good solution.
66
We have the distribution of the water pipe for the chilled water and this is
The Art of the Data Center
Figure 4-3 The MareNostrum supercomputer performs 94 trillion operations per second and
includes more than 10,000 processors.
I was visiting them (the manufacturer) in a facility they have for testing and I
If you could go back and redesign the Data Center all over again, are
there any design approaches that you would do differently?
With the technology today, yes. With the technology today, of course. For
example, we are changing the facility so we will have the capacity to cool the racks
by air or by the rear doors. So we will be having these two possibilities in the same
facility.
What else would I change? Not really a lot of other things.
Are you bringing in more power or is this merely allowing you to make
The Art of the Data Center
In a way you are getting to go back and redesign. A lot of the physical
infrastructure is in place already, but youre once again trying to work
inside this building that you dont want to impact. Has that influenced
how you approach the upgrade?
Yes, absolutely. First of all, I will be thinking very carefully for the scheduling
and the planning for all of the piping and all the analysis, and decommissioning
the old system and installing the new system. That has to be really planned and
scheduled. But of course any new system must be coming with an option for
having the rear-door heat exchanger, because otherwise I cannot host a lot of
them in my computer room. That will be a requirement for all the new computers
in this computer room. Including the disks, because the disks I have right now
are using 5 kW but the new disks we are just installing are using 15 kW. Thats
already a lot of heat that you can get out of a rack.
Did you have to take any special measures to reinforce the area where
the supercomputer resides or was the original infrastructure adequate?
No, that was adequate already. Outside of removing these chimneys everything
else was adequate. So we have continued construction on this ground level floor.
Looking back on the project, were there any surprises that came up
along the way?
Not really, no. Just normal problems. Things that everybody knows but nobody
knows how to fix.
For example, the humidification problem. We have the CRAC units, 12 of
them, humidifying and humidifying continuously because of the change of
temperatures. But if you are humidifying, its very expensive. Its three times the
cost on the power that the CRAC unit is using. If you are not humidifying you
are saving a lot of money. And you can humidify very simply.
I dont know if you live in the mountains or you live in the coast area, but in the
mountains for getting humidity what people do is just have a bag of water on top
of the radiator. This heats the water to the temperature that the water evaporates
and creates the nice humidity. So we have humidity problems in the computer
room and we are fixing it from time to time with bags of water and saving a lot of
money. Because this is just what you learn from nature. Humidification in nature
comes from the sea and with the sun thats getting the water to the environment.
72
publications and web sites as one of the prettiest Data Centers ever
built. During the design phase, how much attention was given to the
look of the facility? Did you consciously focus on aesthetics or did its
striking appearance just happen to come from the design?
Its most likely the second one.
It really comes from the fact that we are installed inside the chapel and we dont
want to change the chapel. We want to maintain the chapel visible. Its not the
computer room that is nice itself, its the sum of the computer room and the
chapel and the fact that we are not changing any of the chapel and we are getting
the computer, the capacity of running in full operation, in this environment.
Of course we designed the computer room for efficiency. So if you have a look at
the pictures you will notice that all the beams of the computer room are mostly
located outside of the computer room not inside and this is to not influence the
fluid dynamics of the room.
We designed this to be 5 meters (16.4 feet) high instead of 3 or 4, because we
want to have better fluid dynamics. This gives us the possibility of matching the
size of the complete box with the size of the chapel, in terms of having the top
of the computer room at the same level as the upper floor of the chapel. And
thats given a dimension and a conjunction of the two bodies, the chapel and the
computer room, which is very good in terms of visibility and environment. It
happens to be that this solution fits efficiency for the computer room and at the
same time the visibility.
It sounds like after the decision was made to build this in the
chapel the design pieces came together as you worked around the
constraints of the building. You didnt necessarily intend for this to be
a showpiece, but by virtue of putting it in the chapel it became that.
Thats correct. When people realized that this is inside the chapel, the architect
had a challenge because he wanted not to destroy the chapel, but to show all the
vantages of the chapel. For the engineer developing the air conditioning it was a
challenge because in the design everything was very efficient, although inside the
chapel, not so much.
73
We wanted to have this visible from every place in the chapel, so we designed
Figure 4-4 The height of the supercomputing centers glass walls were chosen to optimize airflow
and align with the physical dimensions of the chapel.
Figure 5-1 Calcul Qubecs Data Center resides in a concrete silo that originally housed a particle
accelerator. Images provided courtesy of Calcul Qubec.
Chapter 5
Calcul Qubec (Compute Qubec)
Essential Details
Organization: Calcul Qubec (Compute Qubec)
Location: Qubec City, Qubec, Canada
Online: August 2009
Notable features: Server environment is located in circular building that previously housed a Van de
Graaff generator
Time to design and build: 29 months
Size: 2,340 sq. ft. (217.4 sq. m)
Power: 1.2 MW
Tier: 0
Cabinet locations: 56
Infrastructure Delivery: Power and cabling infrastructure are provided overhead. Cooling is distributed
through a ring-shaped, cold plenum, with hardware exhaust vented into a cylindrical, hot core.
Structural loading: 940 lb. per sq. ft. (4,589.5 kg per sq. m)
Fire suppression system: Double-action dry pipe sprinkler system
Data Centers are typically all about squares
Background
The Interview
Please tell me the history of the silo building that houses your
supercomputer.
The building was built in 1965. It was a research center in nuclear physics and it
had a Van de Graaff particle accelerator.
We had to make some relatively minor changes. At first I thought that it was big
changes, but in fact demolishers are very efficient at what theyre doing.
77
The silo was housing the accelerator itself. And it was accelerating particles
We had the idea of putting the computer inside the silo but at
first it was just a crazy idea.
78
How long did it take for you to come up with a design and then to
The Art of the Data Center
beside us. So they will be providing a generator very soon. The basic infrastructure
Tell me about the cooling configuration of the Data Center. Your server
cabinets are laid out on three levels of the silo, facing outward so they
are surrounded by a cold air plenum and there is a hot core in the
middle, while the Data Centers mechanical components are in the
sub-basement.
There is a cold plenum on the outside of the machine. Its an annular, ring-shaped
cold aisle. The cold air comes from the basement, goes up into this vertical cold
plenum, which has a ring shape. The air is drawn by the compute nodes, the file
server, every piece of equipment draws air from this cold air plenum and throws
out the hot air in the center cylindrical hot aisle. And then there are six large
variable drives, industrial blowers, that pull the hot air down into a square room
that has cooling coils on three sides. These are walls of cooling coils, custom-
designed, about 12 ft. (3.7 m.) wide by 8 ft. (2.4 m.) high. Very efficient coils,
four layers. They could absorb up to 1.5 MW of heat. So the blowers they force
the air to go through the cooling coils and then the air comes outfirst the air
goes through filters of course before going through the cooling coilsand then it
exits on the peripheral cold aisle and goes back up again.
I always get the question of why we were pulling the hot air down and pushing
the cold air up. If we wanted to benefit from the natural convection of the hot
air upward we would have to put the cooling system on top. So then we would
have the water on top. It would be more difficult for the maintenance because we
would have to climb all the way up to do the maintenance, change filters and all
that. And also, we wouldnt want the water to be on top of the servers. And the
convection effect is negligible compared to the blowing capacity that we have.
80
Each blower can pull up to 20,000 cfm (566.3 cmm) each, so we have 120,000
The Art of the Data Center
cfm (3,398 cmm) of blowing capacity. These blowers are very efficient because
they are very large and theyre variable drives.
Figure 5-2 Variable drive blowers move air through Calcul Qubecs Data Center at low velocity,
consuming little power.
chilled water at about 4 degrees Celsius (39.2 Fahrenheit). We heat up the water
have shut down only 10 percent of the machine but we shut down 20 percent.
The Art of the Data Center
will have less air than others, so they will start to accelerate their fans because
Any other interesting design elements in your Data Center that people
should be aware of?
The most important thing is that we are using large, variable drive industrial
blowers. The blowers have a diameter of 1 m. (3.3 ft.) so they turn very slowly. I
dont recall their power consumption but they consume almost no power if you
compare that to the blowers that you will find in servers or even blades that have
more efficient blowers. These are order of magnitude more efficient and they are
variable drive. So, we blow the amount of air that the server needs. Because we
have a very large cross-section so that this air can go through and we keep the
84
air velocity very low. Beneath 2.4 m. (7.9 ft.) per second in the worst case. Thats
The Art of the Data Center
if the blowers were running at full speed, 120,000 cfm (3,398 cmm) through
32 sq. m. (344.4 sq. ft.), then we would get about 2.4 m. (7.9 ft.) per second.
And we have pressure differential sensors to measure the amount of air that is
going through the servers and we just replace the air that the server needs. And
because the air velocity is so low the pressure is uniform within the cold aisle. The
problem with air velocity is when it goes too high the pressure inside the cold
aisle is no longer uniform, then there is friction. Friction produces turbulence
and turbulence produces differences in pressure. This is the root of all cooling
problems in Data Centers, is turbulence.
Figure 5-3
Hardware exhaust
vents into a hot
core and then
is drawn down
through the floor
grating to cooling
coils on the
ground floor.
The other factor about our design is that the racks are arranged in a circle. Thats
another interesting detail. Usually in Data Centers when you have problems of
turbulence, of differences in pressure or differences in temperature because of
pressure or because you need to move the air through very long distances, they are
at the end of rows. Sometimes in Data Centers the air will be very cold on one
side and at the other end of the aisle, the end opposite to the CRAC units, then
maybe there will be hot spots. And the hot spots are caused by turbulence and
turbulence is caused by friction and by corners, also. In a circular arrangement
like ours there are no corners.
85
Im curious, how are the noise levels in your Data Center? With
Were there any surprises that came up during the conversion of the
silo building into a Data Center?
We anticipated that we would have problems demolishing stuff. I told you about
the mezzanine and the concrete slab floor that needed to be removed. This is a
concrete slab floor, 2 ft. (.6 m.) thick, armored very strongly. The silo itself has
2 ft. (.6 m.) thick walls of armored concrete. It was a nuclear grade installation
so it was made very, very strong. In the basement the walls are 3 ft. (.9 m.) thick,
armored. We needed to cut some openings in there. We thought that would be
an issue but in fact all of the demolition and cutting out some openings in to the
silo and to the walls in the basement it took less than a month. And we needed
to remove also the metal enclosure of the accelerator. This thing was maybe 25 or
30 ft. (7.6 or 9.1 m.) high and its a metal enclosure that was 2 in. (5 cm.) thick.
We thought it would be a big deal to put that into pieces but in the end it was
very easy.
During the construction there were delays. Essentially it was the subcontractor
that prepared all of the metalwork for the floors. So the beams, the grating, that
took a lot of time. It took several months of delay because the subcontractor did
not do his job very well. He was late. So that was the explanation for the delays. It
took about 4 months more to build than what was expected. We expected about
4 or 5 months and it took maybe 8 months.
86
The Art of the Data Center
Figure 5-4 The housing for the Van de Graaff particle accelerator was sliced into rings to facilitate
its removal. Shown here, the accelerator base rests on the floor.
Surprises? No. We didnt have a lot of cost overruns. We had a few but not too
much. The biggest one is that at first for the fire protection system, we thought we
could use water from the adjacent building. When they made tests they realized
that there was not enough pressure. So we had to cross the street to get the water
for the fire sprinklers. That was a small surprise that cost about $100,000 to drill
through the street. On the campus we have the problem that the water pressure
that we get from the city is barely enough. It depends on when you make the test,
sometimes the pressure is enough and sometimes it isnt. When we made our test
it wasnt. So we had to take measures.
We didnt have any problems with the design. The design works well and we
didnt have any surprises there.
87
How do you bring server cabinets and hardware in to the various levels
of the silo?
Inside the silo there was on the roof, built in to the silo, a movable crane. It was
a very sophisticated crane that was used to lift the housing of the accelerator,
for maintenance. This housing weighs several tons. There was this moving crane
attached to the roof of the silo. So we used that for building. If we didnt have this
crane inside the silo it would have been much more difficult to build, to demolish
what needed to be demolished and to build everything inside the silo.
The crane, it could move in two axesone longitudinal, in a straight axis, and it
could also rotate around the silo. It could move from one side to the other and it
could also rotate, so you could bring the crane at any position. And of course it
can go up and down. So it had three degrees of freedom.
88
If the building people if they did not have this crane for the construction, it
The Art of the Data Center
would have been very difficult for them because they used it a lot to lift stuff
and for the demolition. Like the mezzanine, for instance, they removed it in one
piece. They cut it from the wall and they cut the columns that supported the
mezzanine and they removed the floor of the mezzanine in a single piece using
this crane. It was quite impressive to look at that when they were doing it.
There is an opening also inside the silo. And this opening was existing in the
original silo. It was the opening through which the accelerator was brought inside
the silo. This opening was originally filled with concrete blocks. So it was kind
of closed but it was just concrete blocks that we removed and we put in a very
large window. In fact, the window has the height of the three levels where there
are servers. When the machine was delivered this window was not there. It was
left out. We put it at the end. So, the cabinets were delivered, and put in each
level through this window with the crane. In fact, with a truck with a platform
Figure 5-5 The silos crane, originally installed to lift the housing of the particle accelerator to
enable maintenance, used during demolition and construction of the Data Center. Here
the crane removes a concrete column.
89
lift. Anyway, if we need to bring more racks then we use the crane that is still
Figure 5-6 The original appearance of the silo building, before its conversion into a Data Center.
90
would house a freight elevator and some outside stairs and they rejected it. So
The Art of the Data Center
even if we had the budget they didnt want us to modify the silo because its kind
of a landmark on the campus so we werent allowed to make modifications outside
the silo. We looked at the possibility of putting a freight elevator inside the silo,
but we still would have needed some stairs. So adding the freight elevator would
have used even more space, and then there were security issues, building codes.
Figure 5-7 The metal structure of the three data floors, as seen during the early stage of construction.
91
Essential Details
ORGANIZATION: Cisco
Location: Allen, Texas
Online: April 2011
Notable features: Linked to a second Data Center in the same metro area. Enclosed cabinets with
chimneys. Airside economizer. Rotary UPS. Rooftop solar panels. LED lighting. Lagoon to capture rainwater
for irrigation. Building reinforced to withstand 175 mph (281.6 kph) winds. LEED-Gold certified.
Time to design and build: 22 months
Size: 162,000 sq. ft. (15,050.3 sq. m) total, with 27,700 sq. ft. (2,573.4 sq. m) of hosting space.
Power: Two redundant 10 MW feeds; IT usable capacity of 5.25 MW.
Tier: III
Cabinet locations: 754
Power Density: 6.5 kW average per cabinet, 20 kW maximum
Infrastructure Delivery: Structured cabling, electrical conduits and air cooling delivered overhead
Structural loading: 250 lb. per sq. ft. (1,220.6 kg per sq. m)
Fire suppression system: Pre-action dry pipe sprinklers, with VESDA smoke detection
Whats better than a highly-available, energy
Background
The Interview
What is Ciscos overall Data Center strategy? What is the companys
goals for its Data Center portfolio?
Two things. One is we need to address the business requirements as they pertain
to Data Centers. Number two, showcase Cisco technologies that are relevant to
the Data Center and generally raise the awareness and the mindshare that Cisco
has with prospective or current customers out there.
95
6: Cisco
There are three main pillars that we wanted to address, that are the Data Center
outcomes from some of the business requirements.
One is we have to get the right capacity in the right places in the country, and
actually globally as well, and use that capacity in the right way. So we need to
optimize the use of it and possibly we need to build or partner with people to give
us Data Center space andprobably increasingly in the futureother resources
in the Data Center.
Number two is we need to increase the reliability of Ciscos IT environment in
general so that (if ) theres a catastrophe in one area, a natural disaster wouldnt
significantly influence Ciscos business.
And number three is we needed to change how we provided technology
infrastructure services to Cisco. This basically means moving to a services based
organization and using a lot of automation, understanding our costs really well,
driving down the cost, benchmarking many different aspects of how we provide
services.
Within that strategy, what functions does this Data Center in Allen,
Texas, serve for Cisco?
At a high level they, in conjunction with our Richardson, Texas, Data Center,
increase the reliability because for the necessary applications we can operate
in much more of an active-active way. So we definitely increase availability
and reliability of the services that we provide both internally and to external
customers, because Cisco.com for example is serviced in large part out from those
Data Centers.
LEED Certification
Leadership in Energy and Environmental Design (LEED) is a rating system for the design, construction, and
operation of green buildings. Facilities can achieve ratings of certified, silver, gold, or platinum based on
criteria within six categories: sustainable sites, water efficiency, energy and atmosphere, materials and
resources, indoor environmental quality, and innovation and design.
96
Were raising the tier level of the Data Center so just by itself moving these
The Art of the Data Center
applications and these business processes into a Data Center with a higher tier
level, even doing that increases the availability of the applications.
We are using a lot of Cisco technology and several of the pieces of technology
have new innovations and other features in them that address some of the
capacity and some of the reliability and actually probably some of the way that we
offer our services. Some of the new acquisitions from Cisco that were using down
there in our cloud, for example, specifically address how we provide services. So
I think all three main areas of what we were trying to do with the Data Center
strategy are tangibly implemented in Allen and with its partner Data Center just
down the road.
Figure 6-2 One of the two data halls within Ciscos Allen Data Center. Image provided courtesy of
Cisco. Photography by Cyclone Interactive.
97
You mention Cisco moving to a services model. How did the mission of
6: Cisco
this Data Center influence its design?
We did have discussions about whether we should have lower tier Data Centerswe
built two Data Centers reasonably close to each other, maybe 15 miles (24.1 km)
away from each other, and then used some active-active technology to distribute
the applications between themwhether we needed to build a Tier III plus Data
Center, which the Allen Data Center is, because it would probably have been
cheaper to do that (build to a lower tier).
We decided that in the end it did not make sense to do that because not all
applications are architected to operate in an active-active manner even though
sometimes thats what the users of those applications and business processes
actually need. There were many examples of mostly legacy but even newer
applications that we felt should be housed in a Tier III plus Data Center. There
is a cost difference between a Tier II and a Tier III plus Data Center but when
we looked at the increase we felt that since there were these applications that
couldnt make use of active-active capability, that still needed some higher level of
foundation and an increased level of reliability, so we would go with making it a
Tier III-plus Data Center. Thats one thing.
We also needed to make this a multi-tenant Data Center. There are some business
units at Cisco that wanted their own cage or wanted almost like a mini co-lo
inside that Data Center. Our goal is to provide infrastructure services but at the
moment there are valid reasons why some business units only need Data Center
space, where we provide them network, power, and cooling.
So theres multi-tenancy and its environmentally friendly. We made several design
decisions, many design decisions actually, to make it as environmentally friendly
as we can. Weve got Gold LEED certification for that Data Center, several other
things that weve done like using rotary UPSs for example.
As far as the technology thats being put in the Data Center its obviously a newer
version of some of the old legacy Data Centers that we have and weve used a lot
of the features of mostly Cisco but other partners as well. Technology that gives
us better reliability or is better easily managed or we can automate it better. One
example of this is weve built out one instance of our private cloud called CITEIS
(Cisco IT Elastic Infrastructure Services) is based down there. And that is highly
automated provisioning of infrastructure services.
98
6: Cisco
Figure 6-3 Adjustable louvers allow outside air to be brought into the Data Center. Image by
Douglas Alger.
Figure 6-4 Thirteen air handlers feed cool air into a common plenum above the server rows. Airside
economization allows the facility to be cooled by outside air more than half of the time.
Image provided courtesy of Cisco. Photography by Cyclone Interactive.
100
the area. Can you talk about Ciscos Metro Virtual Data Center concept
and how thats designed?
The idea of this MVDC or Metro Virtual Data Center is that from an application
point of view these two Data Centers look as though they are one Data Center.
However, if there is an issue in one of those whole Data Centers then the
application can recover reasonably transparently if not transparently and still
provide these services that either the application or business process is meant to be
providing.
There are technologies that we are usingsome from Cisco some from some
of our partners, like Oracle especially on the database side, that let us do this.
There are certain requirements on the application as well though, obviously that
the application is architected to support MVDC. But a lot of the underlying
infrastructure and middleware environments also have to support this and provide
the foundation to enable the applications and the databases to be serviced from
either Data Center.
Basically, its a way to provide almost continuous operation if there was an outage
of one of the Data Centers. It is quite expensive to do this from an infrastructure
and middleware point of view so not all of our applications use this capability.
Weve been through a criticality rating exercise with all of our business processes
and applications, rated them and then worked out which ones should be riding
on MVDC and which ones need not be riding on MVDC and then obviously
made those decisions and were well down the path to move the ones that we
think should be on MVDC to MVDC.
The idea of this MVDC or Metro Virtual Data Center is that from
an application point of view these two Data Centers look as though
they are one Data Center.
101
6: Cisco
(24.1 km) apart, is there any concern that both Data Centers
could be hit by a single disaster event?
There is a chance that both will be taken out by a natural or other type of disaster.
We think its very small, however. But there is a chance and therefore we have a
disaster recovery site about 1,600 miles (2,575 km) away. Its a non-production
Data Center so when theres not a disaster all our development and other non-
production use cases go on in this Data Center. Its in RTP (Research Triangle
Park) in North Carolina and if there was a disaster that took out both of those
Data Centers in the Dallas area then we have the capability of bringing up the
business critical applications in our RTP facility, in our DR facility.
The physical distances among the three Data Center locations were
determined in part by the pre-existing location of Ciscos campuses
in Richardson and RTP. Generically, if someone was looking to
implement this model, with two linked Data Centers in a metro area
and a DR facility outside of the area, what are the recommended
distances to have among the three facilities?
The main technology issue is about how far the pair of Data Centers can be apart
from each other given that in most cases you need dark fiber and probably diverse
dark fiber between the two Data Centers. Its not really the physical distance its
the distance of the fiber between them.
My recommendation is to get them as far apart as you can given that you want
to stay within around about 60 miles (96.6 km) from a fiber distance. The reason
why that it has to be of that order from a fiber distance point of view, is because
if you are going to use synchronous write technology on the storage its important
not to introduce too much latency into the application and most applications will
work with a fiber distance of 60 mi. (96.6 km). Some of them start having issues
after that.
102
How did Cisco approach the design of the mechanical system? Texas
The Art of the Data Center
isnt the first place people think of for using outside air cooling, but
the Data Center incorporates air economization and uses outside air
more than half of the time.
For environmental and for cost reasons we wanted to use as little power as
possible in this Data Center. One way of doing this is to use outside air to cool
the Data Center when the outside air is an appropriately low temperature. And
thats just what weve done.
We have a lot of air intake so are able to take outside air when its sufficiently cool
outside. I was pretty surprised, actually, how large the percentage of time is that
we think we can take outside air from in the Texas region to cool the Data Center.
We did raise the inlet temperature in the Data Center as well so that definitely
helped. If its less than 72 degrees (22.2 degrees Celsius) or so outside we can
start taking air in from the outside rather than actually using a specific chiller to
actually do that and using power, so I think thats a great result for us. It saves us
money and as I said it helps the environment as well.
One of the design points obviously is to keep the hot air totally separated from
the cooler air and thats been implemented down there. In that facility we use
chimneys on top of the cabinets and that works really well. Theres very little
if any crossover between the cold air going straight into the hot air intakes
and weve done a lot of testing to make sure that the cabinet temperatures are
appropriate. Weve optimized the inlet temperature into the cabinets so that the
equipment stays within the range of what the equipment maker feels that those
equipment can withstand and keep it within the warranty and maintenance limits
of the device. So we did some optimization there.
103
6: Cisco
Figure 6-5 Enclosed cabinets with vertical exhaust ducts isolate hot and cold airflow, providing
increased cooling efficiency. Image provided courtesy of Cisco. Photography by Cyclone
Interactive.
How does Cisco deal with the fact that some of its own hardware
models vent exhaust to the side rather than following that front-to-back
pattern that is preferred for Data Centers with hot- and cold-aisles?
There are a few answers to that. One is, we continually work with our BUs
(business units) and the BUs are pretty aware of this. There are just some specific
physical limitations when you start getting large switches, with cooling front to
back. We havent been able to solve that issue.
Number two is we have come up with some pretty inventive cabinet designs where
even though it is side to side airflow for cooling we can still use a chimney design.
104
The third thing is that we obviously realize that there are going to be some
The Art of the Data Center
equipment that just wont fit into one of these cabinets. For thatand its not
just Cisco equipment, in fact its primarily not Cisco equipmentwe do have
to understand that some equipment, maybe some storage equipment is the most
common, is going to be out on the floor and its going to be pretty difficult for
us to put a chimney on top of their proprietary cabinet. In that case we keep
those out on the floor. We position them in the most optimal place but the actual
environment is cool out on the floor because were using chimneys and that gets
cooled in a more traditional way.
There are two schools of thought these days around whether or not to
use a raised floor in a Data Center. What prompted forgoing it in this
facility?
With the higher power densities that cabinets are capable of supporting these
days, under floor forced air cooling is becoming less and less the right choice.
Its becoming harder and harder to cool these cabinets using under floor cooling.
Given that and some weight limitations in some of the equipment and some costs
actuallyputting in a raised floor is quite costlyweve made the decision in this
facility and in any future ones that we may build to not have a raised floor. Have
the cabling and the cooling and the power provided overhead. And weve come up
with some, again using our engineers, weve come up with some ways all of those
technologies can be provided overhead and co-exist and I think its worked out
really well.
105
This was the first Cisco Data Center to use a rotary UPS instead of a
6: Cisco
conventional battery-based UPS system. What were the drivers behind
that choice?
It was environmental. Dealing with the lead and the acid from the batteries would
have been more impactful to the environment, so we decided to go with a rotary
UPS. And for the size of this Data Center it will allow us to get the PUE slightly
lower by using rotary UPS.
Any concerns about the shorter ride through time provided by a rotary
UPS rather than what you can get from a bank of batteries?
Not really. I feel as though there are a couple of failsafe mechanisms on those
rotary UPSs. One of them is that in the end if the diesel generator is having
trouble starting, there is actually a clutch that kick starts the generator. We did
some testing of this and we think this is a valid way to go now. I think this will
soon become relatively mainstream, to use rotary UPSs.
Figure 6-6 Cisco employed a rotary UPS rather than a conventional battery UPS system for ride
through power. Image provided courtesy of Cisco. Photography by Cyclone Interactive.
106
If you could go back and design the Data Center again, what if
anything would you do differently?
I would revisit the decision to have chimneys and lay on the table again whether
we should use chimneys or whether we should use warm air aisle containment.
I think thats a different decision. They both get to the same point, which is
keeping hot air and cold air away from each other, but I do think its valid to
revisit that decision.
107
As you say, both solutions get to the same point. What would make
6: Cisco
you consider warm air aisle containment over the enclosed cabinets
with chimneys?
I think you can get better separation. I think you can handle higher density of
heat inside a cabinet using hot air aisle containment. As long as you dont go
overboard on how you design the containment mechanism, I think itll probably
be cheaper to implement that. You do need to make sure that the hot aisle
containment mechanism is flexible enough to allow you to expand or contract the
equipment and the number of cabinets you have in there. I dont think its a slam
dunk but I do think its a better design. Each implementation is slightly different
and depending upon the business requirements it could go either way.
Are there any other lessons from this project that you plan to take
forward into other builds?
The slab decision, I think that was a great decision. I think the density of power
distribution. I think we would spend more time making sure that we dont leave
any stranded power in the Data Center. We may make the power distribution
mechanism even more flexible than it is in the Allen Data Center.
A Data Centers design will obviously vary based on its location and
The Art of the Data Center
the facilitys mission, but generally do you think this facility represents
the baseline of how Cisco will build its Data Centers going forward?
Yeah, I do. Well tweak things. Theres new technology that comes down. More
flexibility about power distribution, for example, is another one but I think in
general yes Im pretty happy with the design decisions made here.
We may make slightly different decisions depending upon the business needs and
other things in the future, but Im very happy with this Data Center. I think some
of the decisions we made were great, working out very well, were the right ones to
enable us to provide the necessary services to Cisco and some of our customers.
So, yes, I think it will serve as the baseline and will be tweaked from that.
Let me offer you the final word. Is there anything else you would like
to highlight for people about this facility?
Please come and visit it. We have a tour in a hall around the outside of all the
Data Center. We can show you a lot of the design decisions that weve made
now, that weve been talking about. We can show you them actually working
and discuss the pros and cons of them. I would certainly have liked to have seen
a Data Center like this before we made the decisions about how to build ours
because at least then you can learn from other peoples mistakes or other things
other people have done right. I would welcome any people, any readers of this
book to come visit the Data Center.
109
6: Cisco
Figure 6-7 A 100 kW rooftop solar panel array provides enough energy to power the office area for
personnel who support the Data Center. Image by Andrew Broer.
Figure 6-8 The Data Center features non-chemical condenser water treatment and an on-site
rainwater retention system for site irrigation. Image provided courtesy of Cisco.
Photography by Cyclone Interactive.
110
The Art of the Data Center
Figure 6-9 Four 20,000 gallon (75,708.2 liter) tanks of generator fuel, four 15,000 gallon
(56,781.2 liter) tanks of chilled water, and one 355,000 gallon (1,343,821.2 liter)
tank of domestic makeup water reside behind the Data Center. Image provided courtesy
of Cisco. Photography by Cyclone Interactive.
111
6: Cisco
Figure 6-10 Server rows in the second data hall are separated by cage walls. Image provided
courtesy of Cisco. Photography by Cyclone Interactive.
Figure 7-1 Citis Data Center in Frankfurt, Germany, was the first in the world to be certified
LEED-Platinum. Image by Christian Richters Photography.
Chapter 7
Citi
Essential Details
ORGANIZATION: Citi
Location: Frankfurt, Germany
Online: May 2008
Notable features: Green roof and green wall features. Reverse osmosis water treatment for cooling.
Harvested rainwater used for irrigation. Rotary UPS. First Data Center certified LEED-Platinum.
Time to design and build: 20 months
Size: 228,000 sq. ft. (21,181.9 sq. m) total, with 100,000 sq. ft. (9,290 sq. m) of hosting space
Power: 14.4 MW from two separate feeds
Tier: III
Cabinet locations: Information unavailable
Infrastructure Delivery: Structured cabling is provided overhead. Power and cooling are delivered under
a 31.5 in. (80 cm) raised floor.
Structural loading: 250 lb. per sq. ft. (1,220.6 kg per sq. m)
Fire suppression system: Pre-action sprinklers
Much as car designers strive to find the sweet spot
Background
The Interview
What prompted you to locate this Data Center in Frankfurt, Germany?
John: We went through a very detailed site assessment process that looked to a
number of factors and at the end of the day those factors pointed to Frankfurt
being the optimum location to construct.
To give you some feeling for what those factors were, they included obviously
risk in terms of the nature of both sovereign risk and physical risk. The second
one was availability of power supplies. The third one was availability of network
connectivity both nationally and internationally, so international network
gateways.
Next was the right sort of construction infrastructure that could actually build
what we were looking for, both technically and also practically. And then also our
ability to support it ongoing in terms of the right level of infrastructure support
both in terms of the market and also from the Citi perspective.
115
Put all those into an equation and you come up with a number of solutions; one
7: Citi
of those is Frankfurt. And we decided to go with Frankfurt because we could
then find availability of the sites and so forth that allowed us to do what we
needed to do.
How long did it take to design and build the Data Center?
Jerry: (It was) quite a fast track program. We started the whole process in 2006.
We had earmarked a locality by mid-2006, but then of course we had to go
through quite a process of approval to acquire the site, et cetera. I can say to you
that really the design process started in September of 2006 and then we went and
had our bid documentation for design, et cetera, by February 2007. We went
out to bid in February/March 2007. We had a contractor on board and on site
in May of 2007. And within nine months we had available to us 12,500 sq. ft.
(1,161.3 sq. m.) of data space and by May of 2008, exactly one year later, we had
the facility complete. So it was a very fast track process.
John: The selection of that site and also the country of location assisted us in
meeting those sorts of timeframes.
Jerry: It did indeed, because with contractors in Germany its different from say
in the States where we also had a quite significant Data Center rollout. There the
procurement of the construction part of the project is usually done under the
construction management type procurement method. In Germany its much more
traditional so we had to have all our designs in place. So it was very critical for
us to get the right team on board. And we did that via a special vehicle where we
had a specialist, a very large international mechanical and electrical engineering
company, and a dedicated construction company and they came together and
formed a partnership for this particular project.
LEED Certification
Leadership in Energy and Environmental Design (LEED) is a rating system for the design, construction and
operation of green buildings. Facilities can achieve ratings of certified, silver, gold, or platinum based on
criteria within six categories: sustainable sites, water efficiency, energy and atmosphere, materials and
resources, indoor environmental quality, and innovation and design.
116
Also in Germany we had the advantage that a lot of the major elements in
The Art of the Data Center
construction theyre very used to pre-cast elements and doing a lot of pre-
fabrication work off-site. And that certainly helped us deliver this project as
well. Again, Germany helped us because thats the particular expertise that was
available to us.
Did you know, as you went through the site selection process, you
would be able to complete the project more quickly by building it in
Germany than other potential locations?
John: Our knowledge of the markets, yes. We had that as part of the selection
requirements because we knew the overall program we had to meet and it was one
of the factors. At the end of the day it wasnt an overriding factor. What was the
most important factor? It was the network connectivity. Second most important
was the power availability.
Jerry: And then of course the question we had to take into account was can we
deliver to our program. And we were confident we could do that because of the
expertise in Germany.
But I would also have to say that we would have done itusing different
procurement methodsif it was done for instance in the UK. I think we still
would have met those same timelines. But it certainly was part of the decision on
location.
and what it has on the overall performance of the facility actually sustainability
7: Citi
can add to the availability, not work against it.
A good example is, if you go down the route of a free cooling solution it can
actually give you enhanced availability because 80 percent of the time youre
getting free cooling and for a significant portion of the time youre getting
free cooling completely which means that youre totally divorcing yourself
from the need for chillers and the associated equipment. So you could actually
suffer multiple chiller failures to a greater level than you would in a more
traditional design and still having no impact on your availability. And thats the
understanding of the overall impact of sustainability.
Now, one of the things that we did do is, and again this is part of the process
when we went out to look at both the design team and also the construction
team, we made sure that sustainability was a contractual obligation of all those
individuals from Day One. Not added on at a later stage or an optional extra,
it was an integral part. So when we selected for instance construction elements,
sustainability, embedded CO2, those sorts of considerations were made alongside
things like performance, cost, et cetera.
Jerry: We had written in to our contract with the main contractor that our
minimum target was LEED-Gold. As you may well know, the contractor has
quite a considerable contribution to make to obtaining the necessary credits to get
LEED accreditation. However, it wasnt just about going for the highest LEED
accreditation, it was in support of our corporate sustainability goals as well. And
as John said it was absolutely embedded in the whole process of procurement,
design, delivery, and construction on site. In addition to free cooling we also had
reverse osmosis, where we save about 50 million liters (13.2 million gallons) of
water per year. Again, there was no interruption in design, this was just part of
the whole design consideration.
John: Even down to the design of the UPSs (uninterruptible power supplies),
we were able to use diesel rotary UPS models, and understanding what that
does in terms of reducing down the amount of cost for medium voltage units,
understanding what that does in terms of reducing down the amount of copper,
for instance. Reducing down the amount of construction that you require for the
building because they have fit a smaller footprint. Reducing down the need for
batteries with the associated issues in relationship to gas discharge and so forth.
All those aspects as well as performance decisions are sustainability decisions. And
so, again, thats how you can get a sustainable and green Data Center that is as
effective with as good availability.
118
The Art of the Data Center
Now, does it at the end of the day use lots of energy? Yes, course it does. Because
a Data Center will do. But things like the integrated design means that you get
a much better PUE than you would likely get with a more conventional design
as well as the advantageous lower environmental impact during construction,
commissioning, and then operation.
Figure 7-2
Citi uses reverse
osmosis to reduce
sediment buildup
in its cooling
towers and saves
about 13.2
million gallons
(50 million liters)
of water per
year. Image by
Christian Richters
Photography.
119
I understand Citi was very particular about the materials used in the
7: Citi
construction of the Data Center and their embodied energy. Can you
provide some examples of how that influenced your choices?
John: We used things like recycled materials in relationship to some of the
cladding systems. We used locally grown timber, naturally seasoned so that
from an ongoing maintenance perspective as well theres no great issues in that
respect. We also looked at things like the roofthe roof actually is a green roof
on there, which again has an impact over its total lifecycle. Because thats the
other key issue, you dont just look at it in the short term you look at it across its
total lifecycle to see what impact it has from an energy perspective. So those are
examples of that type of approach.
Did you use a particular source for defining the embodied energy
values? A challenge with embodied energy is that there isnt universal
agreement as to how to measure it. Also, the embodied energy of an
item can differ greatly by location because whats local to one site
isnt local to another and whats abundant in one location might be
scarce in another.
John: Youre correct, and thats why its very important to have a partnership
with a design company that understand your objectives and understand the issues
in relationship to sustainability as well as having a local knowledge and a local
presence.
We were fortunate in that we utilized Arup Associates. They are a
multidisciplinary both architectural and M&E practice that were also based
in the Frankfurt area as well as LEED accredited. They had LEED accredited
professionals, they have local knowledge, and they also have a great commitment
to sustainability. They have been actively engaged in the Qatar naturallycooled
football stadium, for instance. They are leaders in quite innovative sustainability
designs so that did help us immensely.
Jerry: In a lot of our workshops, that expertise came across and it played a big
factor. I may also say that our contractors were very committed to us attaining
the highest sustainability rating we can. They were very keen also to ensure that
recycled content of the materials were used as much as possible. They had to go
out to their suppliers, their subcontractors, as well and ensure that message was
got through the whole pipeline.
120
And indeed we ended up that the recycled content of the materials on the project
The Art of the Data Center
You mentioned that in the design of facility you reduced the amount
of materials used such as copper. Did you accomplish that entirely
through building a smaller facility or were there other elements of the
design that allowed you to streamline the infrastructure?
John: There were two aspects to it. The cabling aspect comes very much from
the UPS design. You can put in less copper because if youre transmitting voltages
around the building at medium voltage then obviously you need less copper.
What we also did from a data perspective is we made sure we tried to optimize
our cable runs on the data side to reduce the amount of cabling and the amount
of copper weve got. It was important to integrate that in to the cabinet layouts,
in the frame layouts, and so forth. So again youre making decisions with
sustainability being one of the criteria that actually also improve performance in
terms of optimizing the layouts for cooling effect, et cetera, et cetera.
The big message that I always try and stress is if you design sustainability in, not
from Day One from Day Minus One that is, before you start the projectthat
actually it assists you in the overall design process in many aspects and does
not hinder you. You can achieve optimization of that overall design without
compromise to sustainability, performance, reliability, or cost.
Cost is a key issue because if wed gone down the route of a more traditional UPS
solution then that would have cost us more money. We did spend more money
than we would normally have done on the mechanical side but that was offset
by the electrical costs, lower cost from drops. And also lower construction costs
because youre not building big battery rooms with all the issues that go with
them. When you look at it overall it can be beneficial.
121
7: Citi
its design including reverse osmosis water treatment and rainwater
harvesting. The Data Center industry today is focusing on energy
efficiency but water usage hasnt received as much attention.
John: I would point out that Citi is the first bank to have published water
reduction targets as part of a sustainable agenda.
Figure 7-3 Citis Data Center includes a green wall and a green roof that reduce the heat load on
the building and last longer than conventional materials. Image by Christian Richters
Photography.
Your site incorporates a green roof and a green wall and theres a lot of
landscaping across the facility that -
John: I would say theres not a lot of landscaping. Actually, the landscaping was
done as part of our need to provide water settlement. There were two aspects to
it. One, you have to have runoff from a roof and it has to be held before it goes
out to the sewer system simply because in a major flat roof you have a lot of water
coming off there and the sewer systems cant take that volume. So the landscaping
was done because of that.
123
But of course because weve got a green roof the actual roof acts as a reservoir
7: Citi
anyway, so it reduced down the amount of landscaping we did and it reduced
down the amount of earth we had to move to create those. So again the impact
of a green roof, not just from the roof perspective but what is a knock-on effect
from a sustainability point of view was a consideration at the time when we made
that decision. Its offset partially by the fact that you have got a slightly heavier
structure to support it, but again you get longer term benefits because actually
a green roof lasts longer than a conventional roof because its not subjected to
the same deterioration from a UV perspective or from a mechanical damage
perspective. So again sustainability solutions give you a better performance.
There isnt a lot of landscaping there. There is some but not an enormous amount.
I take it then that the solutions used in this Data Center are universal
The Art of the Data Center
Figure 7-4 Implementing diesel rotary UPS systems helped Citi eliminate batteries and other
building materials. Image by Christian Richters Photography.
125
As you look back on the project were there any surprises along the
7: Citi
way? It sounds like a lot of what you did in the room you had decided
early on, but was there anything unanticipated that came up during
either the design or construction?
Jerry: Thats an interesting question. I was trying to rack my brain and see if
I could come up with something. We produced our concept design report and
that was our bible. Any variance from that we had to go through a whole change
control process. Really, looking back there were very few changes.
John: I think that was one of the reasons why we could deliver it in that time
frame. Because the design had been well developed to a point where there was
not a lot of need for change. We did a lot of value engineering and so forth as
part of the design process. It meant that the suppliers themselvesand its very
important when youre doing a lot of off-site fabrication you cant change your
mind once you get to the siteso it was important that we got it right and we
did. And actually if you look at the concept design document and took it onsite
you could navigate yourself around the site very, very effectively.
Jerry: Overall by way of change ordersand this is from memoryfor a job
of this size we were something like 50 or 60 change orders and a lot of those
were referring to locations of some doors and some things to do with the admin
buildings and the ancillary spaces like storage areas rather than the central plant
and the data hall. So there were no surprises, really.
John: The only thing that was a surprise, but it was a pleasant surprise, was it was
a mild winter. That was the one thing that I would say, it was not a hard winter.
Winters in Frankfurt can be very cold, very long, very hard. And we were very
lucky that that year it wasnt.
I realize this is still a relatively young Data Center and you sound very
satisfied with it, but if you could go back and design it all over again
what might you try to do differently?
John: To be perfectly honest I have not given it any thought, and nothing
jumps out at us. Im sure there are now improvements and changes in terms of
availability of materials but I wouldnt change the design team structure or the
construction team structure. I think that worked extremely well. Theres not an
enormous amount we would change.
126
I read that 100 percent of the Data Centers construction waste was
The Art of the Data Center
diverted from the landfill. What special steps had to be taken for that
to happen?
Jerry: I think, again, we had the advantage of being in Germany. Its highly
regulated anyway. Where you have waste you have to separate it on site. So we
had a set of requirements setting that out. Our original task was something like
50 or 70 percent to be diverted from landfill and we got up to 100. And we got
up to 100 simply because we made the commitment to maximize and we had our
specifications and our documents in place and the contractual obligations there
on the contractors and subcontractors.
For anyone starting a Data Center project, can you offer them some
advice?
Jerry: A big advantage for us was having an integrated team. We were very
conscious at the very beginning that we needed somebody who could provide
the architectural, the mechanical, the electrical, the security, and indeed IT
advice as well, and risk assessmentall that capability and disciplines within one
firm. That was a big advantage for us. We were very clear on our coordination,
very clear where the responsibility laid. We had the advantage of this particular
consultant and this is the way we wanted it. They had all the necessary disciplines.
I think that helped tremendously.
And also as I mentioned before, having the concept design report and having that
as our bible that was very, very critical. Because for us much of the early design,
the concepts and requirements, were put together here in London and we worked
with our consultants here in London and then of course we had to transfer at
a certain stage back to our colleagues and indeed transferring design over to
Germany as well. The same firms, the same groups involved but we then had the
language to contend with as well. Because all bid documentation, drawings, et
cetera, then had to be in German. But having an integrated design team was a
great advantage.
John: I would reiterate that. That is the biggest benefit I think. It was a team and
we were able to integrate then our requirements including the sustainability from
Day One, which I think also was a very key point.
127
Are there any final lessons that you would like readers to take away
7: Citi
from this project?
John: From a personal point of view, I think the lesson is integrating the team,
integrating the sustainability alongside the other requirements of a Data Center
performancereliability, availabilityand that you can achieve each of those
and each of those are mutually enhancing rather than working against each other.
Thats the message I would get. If you approach it right and early enough you can
get benefits in all of those areas and its not the detriment of one or the other.
Jerry: The only thing I would add is that it involves the entire supply chain,
down to your contractors, down to your subcontractors and suppliers. They have
a big role to play as well.
Figure 7-5 Citis Frankfurt Data Center at night. Image provided courtesy of Citi. Photo by
Steve Kay.
Figure 8-1 The Lakeside Technology Center in Chicago is a registered historic landmark and houses
multiple Data Centers. Images provided courtesy of Digital Realty Trust.
Chapter 8
Digital Realty Trust
Essential Details
ORGANIZATION: Digital Realty Trust
Location: Chicago, Illinois
Online: First constructed from 1912 to 1914. Converted to telecom use between 1998 and 2000.
Acquired by Digital Realty Trust in 2005.
Notable features: Designated historic landmark building. More power capacity than Chicago OHare
International Airport. One of the worlds largest carrier hotels. An 8.5 million gallon (32.2 million liter)
brine tank for thermal storage. Includes a 20,000 sq. ft. (1,858.1 sq. m) data hall that was first in the
United States to be certified LEED-Gold.
Time to design and build: Varies by data hall
Size: 1,133,391 sq. ft. (105,295.5 sq. m)
Power: 102 MW
Tier: Varies by data hall
Cabinet locations: Varies by data hall
Infrastructure Delivery: Varies by data hall
Structural loading: 250 lb. per sq. ft. (1,220.6 kg per sq. m)
Fire suppression system: Dual interlock pre-action system with above and below heat and smoke
detection
When design began in 1912 on the RR
Background
The Interview
Lets start with some background about the Lakeside Technology
Center. What can you tell me about the history of the building and
how it came into the possession of Digital Realty Trust?
The site was the headquarters of the RR Donnelley Corporation and Mr.
Donnelley had his office there. If you ever get a chance to go to the building,
theres this incredibly beautiful library and that library housed all the first edition
books that they printed.
131
Its got this cool industrial story. He was one of these guys who was an industrial
Those guys had some success. They leased a couple floors. At this point in 2000,
The Art of the Data Center
2001 two or three of the floors were leased and enclosed, but there were whole
floors like the fourth floor or eighth floor that were open. They were open to the
elements and they werent contained so the chilled water plant that ran through
there had to act like it was exposed to the outside air. So it was a 30 percent
glycol plant for freeze protection. The building wasnt in very much pretty shape,
although it was pretty functional.
They got an offer they couldnt refuse from El Paso Corporation. At the time
El Paso was building a fiber network inside of natural gas lines. They had raised
money; they were flush with cash, so they bought out the original developer. Then
as El Paso flamed out and the dot com business wrapped upit stabilized, it had
some cash flow but it wasnt really healthy.
In the time when El Paso was wrapping up, as Digital was growing we had a
funny conversation with El Paso and they said We intend to sell you this building
but were busy getting rid of billion dollar assets. After we get rid of the billion
dollar scale assets well call you when were ready because were really busy. So we
waited patiently and courted our target and over a period in early 2005 we were
able to conclude the transaction and we acquired the building in June 2005.
It was something like $150 million. It was a good price for both of us. They were
able to get a good rate. They lost quite a bit of money on it but at the time there
was not much going on. We didnt have to overpay. We got a fair price for the
income that was there and lots of surplus space and assets. It was really a ball of
wax. We had to spend a lot of time figuring it out.
So in 2005 we took over the building. We started some basic rehabilitation,
putting in better security, just the kind of thing that a professional building owner
does. El Paso was never a real estate company. As soon as they bought it they were
in, generally speaking, financial trouble. So they werent investing in the building.
We started the basic investment program and started to market it for lease. At
the time there was an Equinix Data Center in there, the Chicago Mercantile
Exchange was in there, Qwest had a cyber center and a POP there, so it had
pretty good density and it was pretty well built out. But there were still two and
a half or three floors of vacant space so we knew we could do some other work
with it.
133
We started by rehabbing the base building and then we went around and we
found a number of Data Center telecom spaces that had either gone bankrupt
or reverted back to the buildings control and we did a lot of surplus work. So,
we would take a 500 kW, DC powered, 50 watt per foot (538.2 watts per square
meter) telco space. Generator was good, switchboard was good, cooling loop was
good, density was wrong, power was wrong. We would rip out the DC plant,
put in a UPS (uninterruptible power supply), re-engineer the cooling, give up a
bunch of space. We would bring to market a 75 watt per foot (807.3 watts per
square meter), non-raised floor Data Center space but we did it very inexpensively
so it was priced really attractive and we got great customers. We started to do
some of that where we would rehab the building and this is where we developed
our skill set, taking these kind of scary situations and doing some really, really
hard engineering and turning lemons in to lemonade.
134
Thats a cultural trait that Digital has thats really important to us. Were a real
The Art of the Data Center
LEED Certification
Leadership in Energy and Environmental Design (LEED) is a rating system for the design, construction, and
operation of green buildings. Facilities can achieve ratings of certified, silver, gold, or platinum based on
criteria within six categories:sustainable sites, water efficiency, energy and atmosphere, materials and
resources, indoor environmental quality, and innovation and design.
135
And then we had this sinking feeling of, uh-oh, what have we done? Have we just
that was the revolution that we created for ourselves, really for
our teams and our clients: sustainability is important.
136
windows, so much of the building architectures have an inner wall that allows
The Art of the Data Center
access to the exterior windows. Weve got this kind of double pane effect just by
the need to be able to get to windows. Because of their age and their materials the
thermal expansion properties of the old glass are poor so they frequently break in
the hot and the cold from contraction. We have a big budget to replace windows.
Well, that worked very well for daylighting because it wasnt like we were putting
a single pane of glass in the UPS room. We were putting in a double set so that
you had some containment. If you had a window break you had a whole other
window. So, there were some architectural things that played in. And that was
one of those challenges that drove us crazy. Weve got to put in two windows?
No, thats expensive! We dont want to do that. But in the end, the form follows
function idea played well into that. Where it made sense architecturally, we
preserved the historical look without having to replace the exterior windows
and we got daylighting in. And it wasnt a completely insane (scenario in which)
theres a single pane of glass between the outside world and the UPS room.
Figure 8-3 Digital Realty Trust installed a second layer of windows behind the originals, providing
more physical security while allowing natural light in to certain areas.
137
Have you used those for cooling or are you routing physical
The Art of the Data Center
Figure 8-4 Architectural details from the building feature symbols from the history of printing.
140
For example, at first this window budget was killing us. As we stabilized the
The Art of the Data Center
building and grew it we were able to meet with the city and find a solution where
we have to buy a fairly expensive window but it matches the look and feel of the
historic windows. So, slowly but surely weve been replacing those. So, those are
the main things.
This is one of those buildings where normally the traditional real estate parts
of the buildingthe landscaping and the toilets and the parking lotare very
simple and then the Data Center is complex. This is a building where the Data
Center is complex and the building management is complex, because of things
like the maintenance of the faade. There are a bunch of little architectural
features embedded in the concrete in the window sills. Sometimes those fall, so
youve got to be careful from a safety perspective that that old stuff is maintained
and youre not adding risk to the population, and then the look and feel of those
things have to be right. So the faade is really one of the dominant pieces.
Then there are a number of historic lobbies and common areas. Those we really
just have to maintain this incredible, beautiful walnut woodwork and sandstone
carvings. And so those are places where were just off limits from development.
The management offices for many, many years were the original; Mr. Donnelleys
corporate office was where the general manager sat. Weve since leased that to
customers who love it. Weve built some more traditional functional office but
there are sections of the building where we go, It is what it is. We cant take any
of that wood out. We cant reroute the electrical. We can do some wireless LAN
but theres some historic office that weve leased to customers and its going to be
historic offices for the life of the building. But we knew that going in, so it wasnt
a place where we said If we could rip out all of this office we could put in a Data
Center. That fit into our master programming as theres going to be some office
there.
So, theres certainly an element of that. But those are benefits, right? Those are
attractions to the building. They get a lot of press. People love to see them. In
fact, this library I mentioned is common area in the building. So any of our
customers in the building can check out the library for meetings and events, its a
big conference room and its beautiful. Thats a nice feature. Im sure the property
manager and the guy doing the economics would like to be doing something
different with it, maybe it costs too much to clean. But from a branding and a
product part of the building its a huge positive.
One issue that has come up is, when we first did the development there was
nothing down there. Well, as Chicago has grown and the economy has recovered
that area has become gentrified. We have all the proper permits and licensing for
141
all the equipment therethe generators and the equipmentbut weve gotten
Figure 8-5 Once abandoned, the building was converted to telecommunications use beginning in
1998 and has since been fully restored.
142
So there wasnt anything specific to 350 Cermak that hurt us. There were some
I normally ask about power density and how many cabinet locations a
The Art of the Data Center
given Data Center has to offer. I assume that varies among the various
data halls within the facility.
It does, but since I was involved from the acquisitions due diligence to the master
planning to the final build out I can tell you with pretty good certainty. This
buildingremember, most of its programming and most of its large footprint
customers moved in in the 2000 to 2004 timeframeso outside of one financial
services company that owns almost a whole floor, much of the space is between
50 and 100 watts a foot (538.2 and 1,076.4 watts per square meter). Theres
not a lot of ceiling height here so theres a decent proportion of the suites that
dont have raised floor so youre naturally limited to 75, 100 watts a foot (538.2,
1,076.4 watts per square meter).
The newer builds, Digital has built up to 200 watts a foot (2,152.8 watts per
square meter). One of the clients in one of these very high density 25,000 to
30,000 foot (2,322.6 to 2,787.1 square meters), have built up to 400 watts a
foot (4,305.6 watts per square meter), but I would guess the building average is
probably 75 (538.2).
of leverage on those folks. I imagine this funny scenario where theres a battery
The Art of the Data Center
technician somewhere in 350 Cermak and he never leaves the building. We just
send in tacos and food and sandwiches and hes just doing battery maintenance in
my building forever.
We do see a lot of those advantages of scale with the maintenance guy for UPSs,
for spares. In that building I happen to have one more than one UPS type, but its
only two. I have a Liebert set of spares and I have an MGE set of spares. Im able
to take that spares kit and spread it across 10, 15 MW of load. So there are a lot
of efficiencies at this scale.
On the headache side, certainly on the chilled water plant it keeps people awake
at night. A big common system is risky. One of the things about when we build
our Turn-key Data Centers, theyre usually 1125 or 2250 kW, we have dozens
and dozens of those systems so if any one of them blows up and fails our fault is
contained. We dont have our whole client base exposed to a common system. So
any time we do things at large scale with big common systems it makes us a little
bit nervous. Our competitors build bigger systems and we think thats risky and a
lot of our customers agree. They like that they have these smaller, discreet things.
Until we did the complete upgrade of the chiller water plant across the street that
was my nightmare scenario. Something happened at the Cermak cooling plant
and we have to turn most of the building off. Now that weve been able to invest
more than $10 million in the infrastructure of that facility I can sleep comfortably
because its reliability and its availability and its design are just like everything else
we have, so its neutral. So thats certainly one (challenge).
At that scale on the power grid, you start to have higher profile and bigger
Figure 8-6
The Art of the Data Center
Employees and
customers enter
and exit the 1.1
million square
foot Lakeside
Technology Center
more than 2,000
times per day.
149
Since obtaining the facility and doing various buildouts and upgrades,
right price for it. We were able to invest more capital and grow it. Now its worth,
The Art of the Data Center
Not that most people are going to fall into this sort of Data Center
I have seen a lot of Data Center projects over the years where the
The Art of the Data Center
I said, Dont ask. Ill tell you later, but just go do it. So they did it and what I
Let me offer you the final word. Is there anything else people should
know about this site?
At the risk of inflaming the marketplace, because this is one of these image
issues we have: Data Centers consume a lot of power, is that good or bad? My
argument is Data Centers are huge engines of productivity increase. Data Centers
using a lot of power is good for the world.
But I think the takeaway is, this building and its recent load growth is now the
largest power consumer in the city of Chicago. That for us is a good milestone in
that were helping the city of Chicago transform from its former industrial base
into its new industrial base built around service and technology. And financial
services are a big part of that, because thats a big client base there. That I think
is a good anecdote and a good milestone for the building. We passed OHare
Airport in the last year.
I do think its elegant, the function of this building had in the past
and the one it has now. It seems very evolutionary.
From an economics perspective, like what is redevelopment, were taking capital
and materials that came from the Industrial Revolution and were reusing them in
this revolution. Its a cool, cool circle that were in.
Essential Details
ORGANIZATION: eBay
Location: Phoenix, Arizona
Online: October 2011
Notable features: In-building data hall, containerized Data Centers on the roof. Air and liquid cooling to
cabinets, outside air cooling, adjustable computing capability.
Time to design and build: 22 months
Size: 42,000 sq. ft. (3,901.9 sq. m) total, with 14,000 sq. ft. (1,300.6 sq. m) of hosting space in the
building and rooftop containers.
Power: 4MW for IT initially, expandable to 6 MW for the in-room data hall and 6 MW for containers
Tier: II
Cabinet locations: 256 on the main floor; roof can house 10-12 containers, whose rack count can vary by
configuration
Power Density: 14 or 28 kW per cabinet
Infrastructure Delivery: Electrical conduit, structured cabling, and cooling are delivered overhead.
Liquid cooling piping is routed below a 6 ft. (1.8 m) raised floor.
Structural loading: 750 lb. per sq. ft. (3,661.8 kg per sq. m)
Fire suppression system: Double action, pre-action interlocked system. VESDA detection system.
When online marketplace eBay built its Project
Background
The Interview
Project Mercurys design came about in a unique way, with eBay
specifying requirements and then having design firms submit
proposals. Tell me about its development.
I joined eBay in 2009 and they were right in the middle of Project Topaz, which
is a Tier IV Data Center built in Salt Lake City. I took on that project kind of
mid-stream, and then brought that to completion which was in May of 2010.
159
Tier IV, they did a lot of innovative things in there. They got to a PUE design of
9: eBay
1.4, so they elevated the temperatures. Theyve done a lot of the best practices to
go back and get the best efficiency. Theyve gone to 400 volts, theyve done a lot of
things in the industry. They did a lot of best practices and it will serve our needs.
But the problem is, it still had a lot of the same older approaches in it.
There were fixed UPS amounts per room. There were three 20,000 sq. ft. (1,858.1
sq. m) rooms. It was expected to have lots of space that you could spread out. And
so the philosophy of space is cheap I should build that up and I can just spread
things out as I need. It was a lot of facilities-focused decisions in that. And what I
found was everything was also based on a Tier IV expectation. So, the applications
no one had really done the inventory to find out what has to be in a Tier IV Data
Center and what could be in a lower tier Data Center. So that was the first thing I
did was just go back and assess that. And what we found is up to 70 or 80 percent
of the applications that run in our Data Center, the equipment that actually runs
in the Data Center for those apps, can be in a Tier II. So were spending a lot of
money to basically do overkill on a lot of applications that are resilient already.
That was now where we said lets right-size applications. That was the first
philosophy.
I work in an organization right now where all costs are included. So I own all of
the Data Centers, the networks, supply chain, security, etc. My peer owns all the
provisioning. He takes the cache equipment we have and he basically creates the
infrastructure as a service to deploy out whatever we need for whatever app. So you
profile equipment, you can now make a decision and push things out. And then my
other peer is the one that does site operations and analytics. Its really looking at the
health of the site and the uptime and then all the data around that and presenting
all that information back.
What we found was, lets filter through these apps and from there lets figure out
what we should put in the Data Center. The other part was, we dont know really
what we need when we need it so we need to build a generic, modular approach.
Thats one of my background piecesmodularity is really important in all of this.
And modularity at almost every level, to be able to scale out components as you
need. And multi-tier. Since we had a Tier IV Data Center how would we augment
with a multi-tier center next to it. Dont lose your investments that youve already
had. Lets figure out how we could build, utilize the Tier IV to give it more runway
but now take out things to free up capacity in the Tier IV and put them into
adjacent Tier IIs that utilize the same power infrastructure, that utilize the same
security teams, support teams and delivery channels, and all that on the same site.
160
After that then we started really looking at well what are we putting in the
The Art of the Data Center
Data Center. And (there were) tons of skus, tons of configurations, lots of
people making the choice about what hardware is going in there and not really
standardizing. Just earlier this year we looked at how we could reduce the number
of skus and have standard skus. We took 15 of the top skus companies can have
hundreds of them; we had well over 100and we were able to get those down to
two. One for was Hadoop, which required a lot of local storage, and the second
one was for everything else. We got common CPUs, common memory, common
bus, common network speedsall those pieces so now we could say Well maybe
we can do front end for search or a pool or any other kind of application could
run on this equipment.
With that, then we started to apply that to the supply chain and we started to
find the sweet spot. So now weve got standard skus how do we package them?
The packaging was really, for me, once we started handing off to the vendor to
do a thing called rack and roll and thats where we give them the spec and the
configuration they need to do. They acquire the components, they build, they
rack, they cable, they image, they test, they validate and then they ship us a rack
that we basically plug in the power and the network and then turn up.
When you start to look at that, the efficiencies that we get of now streamlining
the supply chain process and the amount of volume we start to get because we
have all the same type of hardware being ordered and then the competitive RFPs
we can put out therelots of people competing at it and we get great pricing. So
that was the next piece was consolidating those skus.
Then, what we found is that the sweet spot as we started doing the calculations
of running this workload at this much wattage, there are two valuesand theyre
14 kW and 28 kWthat we want to go after. So its a 48U rack, its up to
3,500 lbs. (1,587.6 kg) In our one config its 96 servers in a rack. Weve got the
ability now to put in multiple failure domains from a network perspective. Weve
got the optimized power strips. Weve got the optimized cooling. All of those
components fall together that we have this unit and that unit now is what we can
order from the vendors. And that 28 kW really works in to I can deliver a lot in
less. I have less racks I have to do. Because of the co-los were in, were paying rent
in a footprint based on the density. And so if I get lower density it sounds like it
would be cheaper but the problem is youre getting more racks to do that same
work and youre actually paying more money.
161
9: eBay
Figure 9-2 Greetings program. Visitors first encounter the eBay Data Centers Tron theme at its
entrances.
162
So when we looked at that total cost we decided that the sweet spot here was 28
The Art of the Data Center
kW and 96 servers in a cabinet. Or 48 servers, for sku two, which is the Hadoop
one. And then we started pushing that. Then we were designing the facilities to be
able to handle any load, but really the scale is that every footprint should be able
to scale up to 28 kW over time.
Then we start looking at what were building in Phoenix. How do you build
the maximum density inside of a Data Center that can scale both on the
infrastructure side and then as youre doing tech refresh and rolling equipment
in and out? You can replace it with footprints and double them as the business
needs it.
Now all those pieces started coming together. And then we said I want free
cooling in the Data Center. I want it in Arizona. Its one of the hardest places
for us to solve this. We put the public RFP out there for the industry to take
these parametersfree cooling in Arizona year-round, modular, multi-tier Data
Center solutions, and ultimate flexibility: we want to be able to have racks or
even containers on the roof. And so we did this in a small footprint. We were
building the building already, and we said lets build up as a Data Center adjacent
to our current one on the land we have. Thats where we started saying 3,500 lbs.
(1,587.6 kg), with containers and everything else. We need 1 million lbs. (453.6
mt.) of steel in the building. Were basically building a 10-story building in 3
stories.
But once we had all of those parameters together, we really figured out the
optimum way from a cost standpoint to build a multi-tier center. Because weve
got the Tier IV next door to it.
The challenge out to the industry, this is what really worked with the public RFP
process. The design and consulting engineers out there, the architecture firms, the
engineering firms, they love this. Because it wasnt a company like eBay coming
out and saying this is how you have to build the Data Center. It was Heres our
business problem, here are the components that we really need you to design to.
Now you tell us how to design the Data Center. And that worked. That worked
really well. We had some great innovation and the guys really thought about it.
It took away a lot of the constraints they had in the past and came at it from a
problem statement.
And what we came back with, we finally chose EDI as the winner and they
brought in Winterstreet Architects and AHA Consulting Engineers and between
those three they had a really flexible, simple, efficient, and modular design. And
then when they brought that back in we saw the concept and said Ok, now
163
weve got to blow that up a little bit because our demand just doubled. Heres the
9: eBay
footprint which you have to play with. Youve got a little more space. Whats the
max you can fit in there? What can I fit in that building? Whats the optimum
capacity I should put in? And then I want to grow it in 2 MW chunks over time.
What we ended up with, in 14,000 sq. ft. (1,300.6 sq. m), was the ability to scale
from 4 MW to 12 MW of IT capacity. If you start to divide that out with the
amount of footprint in 14,000 sq. ft. (1,300.6 sq. m)half of its on the roof in
containers, half is in rack and roll on the ground floorit ends up being about
28 kW a cabinet at the end. So its funny how all these things started coming
together with the amount of space we had to be able to do that density. And then
when you look at the reality of what technology is doing, today the containers are
landed on the roof, we found the sweet spot of about 450 kW because that is the
maximum airflow that you can actually get across the equipment in that area of a
contained box, to be able to continuously cool 28 kW. So we found the ceiling of
air. Great.
On top of that we said we wanted free cooling year-round. So we went and took
the hottest day of the year with a wet bulb and that was 85 degrees Fahrenheit
(29.4 Celsius) and they designed a hot water cooling loop. And the hot water
cooling loop is a supply of 87 degrees (30.6 Celsius) because thats eventually
what it gets to at the server. But 87 degrees (30.6 Celsius) were also looking at
being able to plug in, not just todays air-cooled stuff but the future. I want to be
able to have liquid cooled right to the chip. Because 87 degrees (30.6 Celsius) to
the chip is cold.
So, you see all these pieces are starting to come together. Were building a very
flexible Data Center that allows us to pick the tier level, pick the density as
the business needs change. And they will change, and they did change literally
overnight here. And as we see over the life of the Data Center weve given
ourselves runway. Todays best air-cooled solutions Ive got to the cap. Im
maximizing my floor space, Im maximizing my supply chain, Im maximizing my
efficiency.
Because containers that were putting in there, some are using 87 (30.6 Celsius)
as the primary source, the other ones are using outside air and just regular swamp
cooling and theyre getting free cooling the majority of the year round. Ive got
PUEs of less than 1.1 at 28 kW density in cabinets. Thats the kind of result we
can get, that our overall cost per MW is going down because of all these variables
that came together to build for what the business needs versus building what the
Data Center needs and then just accommodating the equipment afterwards.
164
Thats been the lesson really for us. Weve created this multi-tier modular strategy
The Art of the Data Center
that allows us to scale. By the way, the RFP process we did for the Data Center we
followed with the actual compute equipment. So we put out that were looking
for 96 servers in the rack, 48 servers in a rack, for these two skus. Were looking
for these CPU and memory configs and network configs, etc. So we gave them
all the specs. Then we said youre going to be scored on your unit costs delivered,
because theyre going to package up and do everything else. Youre also going to
be scored on the wattage consumed with our average workload over its useful life.
So we took the depreciation cost of the server plus the wattage costbecause I
pay that billfor that equipment over its 3 year life. And we made the decision
on the IT equipment based on that. So, all-in TCOa real TCO. We made a
vendor decision, an IT decision. I was really pleased with that. And that yielded
great results because the first round went to one vendor that was a surprise
to everybody because they had the most efficient design. And then the other
vendors saw that theyd lost and we told them why they lost and they tuned their
equipment and a different vendor won the second round.
So Ive got two vendors on the roof and two vendors on the ground floor, Day
One, opening this Data Center in Phoenix. And theyre both uber efficient.
Theyre both highly cost effective. Theyre both dense. And theyre both getting
free cooling almost year-round.
9: eBay
have to ship all over the world so they ensure that their operating temperatures of
the chip are going to work everywhere. Which means there has got to be 20, 30,
40 percent overhead of temp, right? If youve got a really well tuned environment,
you can raise those temperatures and stay within those boundaries.
Through the Chill Off 3 (a head to head comparison test of Data Center cooling
products) we tested workload with both air and liquid. And we found out that
there was no loss of performance at 87 degree (30.6 Celsius) inlet air. We found
that there was a shadowing removal, meaning that the chip with memory in front
of it and a CPU behind it the memory will heat up the chip further. Thats from
an air standpoint. But even with the shadowing its a little less efficient but it still
didnt have any errors and the boxes were running fine. And then when you put
liquid to itcold plate was one of our teststhe shadowing goes away so the
efficiency gets even better. And then at that point you can now look at the chip
and say Am I really optimizing that chip? Because what I want is a gas pedal.
I want to be able to raise the frequency of that chip or decrease the frequency
of that chip just like I would have a Prius turns into a Ferrari, because when we
need it we need it. I dont care if Im paying more wattage for it, it just needs to
stay within the thermal boundaries, which we can if weve got liquid to the chip.
Boomwe can burst and use less machines to do the work and then back off in
the times when its not loaded. So, you take that parameter, too.
Figure 9-3 Illuminated panels at the end of each row reinforce the eBay Data Center's futuristic appearance.
166
Part of the RFP was we want direct access to the CPU to be able to adjust the
The Art of the Data Center
voltage so that we can adjust the frequency of the chip as needed, so our engine
can go back and adjust itself. So, weve got that hook in there. At the same time
we now said build out the Data Center that I can roll in a cabinet thats going to
use air, side cooling, or a passive door, or any other solution around it, or direct
liquid to the chip, without having to redesign the Data Center. So we delivered
two loops, the 55 degree (12.8 Celsius) loop and the 87 degree (30.6 Celsius) hot
water cooling loop to every location in the Data Center. So the containers can use
it, the racks can use it. I spent maybe an extra half a million dollars in capital to
be able to do that Day One and that depreciated over 30 years is peanuts in what
were doing.
So here weve got the capability now to do from 5 kW to 40 in a footprint and
an average scaling up to 28 kW a cabinet through every rack in the Data Center.
So the useful life of a Data Center just extended. And the capabilities for us to
go back and replace equipment thats air cooled today and roll in a liquid cooled
one in the exact same footprint and get potentially two to three to four times the
performance? Amazing. And I spent half the money that I did before.
Whats the break point at which you want to change from air cooling to
liquid cooling?
Anything over 20 to 25 kW. Youre going to pay a little bit more for the liquid
side. Again, there are products coming. Were pushing hard because they know
what our workload is, they know what were trying to accomplish, and if they can
get that CPU variation its a huge win on the TCO. Its not about power saving
mode, where you turn on power throttling and those kind of things. Thats in
environments where its just too hot. What if you have it optimized? Well you
should now be able to go back and to tune it to what you need and when you
need it. To me that is such a difference from what it was in the past.
If you start to get your arms around what compute workload is going to be on
there and then you start to roll in say, if I use liquid cooling I can overclock these
systems for three years because theyre really optimized for that environment. So I
167
can buy cheaper processors and use them like more expensive processors and still
9: eBay
get the same performance. The unit cost may be about the same, but then at the
times when I want to use it Im turning it into a Ferrari. So I may use 120 percent
of the wattage during that four hour period. But imagine at that point you back
off. For the rest of the 20 hours I have less than that. And at other times I can
either go idle with them or I can take them out of commission because I have less
load than is needed.
So Ive got a variable here, a gas pedal that I can go up and down. And instead
of this constant in the Data Center from a load perspective it starts to vary. So
the overall cost of running that equipment goes down substantially when you
have control over the chip, when youre doing it with that temp. And everyone is
going after this from an air perspective: we need to use outside air. Ok. But what
I find is a lot of people are designing Data Centers with the equipment spread
out. Theyre using the cheap processors and more of them. So theyre using twice
the amount of equipment to do the same workload in a cheaper building to get
a PUE reduction of another .1 and theyre missing the point. Theyre still using
twice the amount of equipment. Theyre like Well, the wattage is two-thirds of
what it was before. But youre still using twice the amount of equipment. Youre
using more wattage than I would in a high density, high temp solution than what
youve got in yours.
So its a holistic view into the Data Center. When you really know the workload
The Art of the Data Center
and the facilities infrastructure and you get those together and you start to tune it
like a machine you get incredible efficiencies from that.
the outside air and build a Data Center that can manage both. And if you did
9: eBay
that in all the regions? You know, the hardest one is going to be Singapore because
of the humidity. But if you have liquid cooling direct to the chip in Singapore,
youre going to get free cooling.
My point is: these practices about building a modular Data Center that is
scalable over time, that does it up frontnot everything but the ability to scale
in chunksapplies to almost every Data Center out there. Its about how youre
investing in your center. And if the centers are supposed to have a minimum life
of 10 years then you should assume youre going to have a minimum of two to
three tech refreshes. And others could have up to five. Like, we do tech refreshes
every two years on a lot of our equipment because were really pushing the
boundaries here from a performance standpoint. But even if you did it every 5
years you have the ability now to double the capacity. If you could roll in a rack of
the next stuff and not have to rebuild your Data Center and all the complexities
around that thats a pretty big win for the company. And youre not spending 2x
at the beginning, youre spending 15 to 20 percent more on piping and other
infrastructure. Its just proper planning.
from that is here is one of the leading indicators of whats coming. So if theyre
The Art of the Data Center
able to roll that into their Data Center and take advantage of it theyre using half
the equipment or less to do the same work. You talk to an IT person and say you
can spend half the capital to grow in here if you use this solution, like liquid to
the chip. But if they have a Data Center that cant take advantage of it, theyre
going to spend more in capital to enable it in the Data Center than its worth.
Ive said this for years: its the organizational alignment. When you have these
facilities teams and IT teams working together, and thats exactly I believe the
secret sauce at eBay, Were all in it. Were in the same game, we have the same
goals, we have the same budget. We all have to go lower the cost per megawatt
to deliver capacity for the actual company. And that works out to be watts per
transaction, or watts per user or watts per whatever variable. Its still watts. And
the less watts I can use to do that and all the other knobs you turn for itthe
efficiencies of the servers to the Data Center efficiencies to liquid cooling to air
cooling, whateverthey all basically lower the amount of watts per whatever.
How are you delivering the infrastructure into the Data Center? Is it a
raised floor environment?
Yeah. This is whats ironic. I didnt like raised floor before because it was kind of a
waste. I dont need to have raised floor to deliver cooling, right? Modular cooling,
I can put in sidecars or overhead, etc., is pretty easy. But what I found in this
environment is it was worth it to put in a raised floor and we have an almost 6 ft.
(1.8 m.) raised floor.
I know, its ironic, Im building one of the biggest raised floors out there and I
dont like them. But the point is, theres no air underneath it. Everything under
that is pipe work because weve got the two loops delivered to every one of the
locations so that now I can snap in either a sidecar air conditioning unit or a
passive rear door or I can plug directly into the rack as those liquid racks become
available.
171
9: eBay
Figure 9-4 Liquid cooling piping is routed below a 6 ft. (1.8 m.) raised floor while power, cooling,
and cabling are distributed overhead in eBays Project Mercury Data Center.
In our next tech refresh I am pushing really, really hard that we are going to have
liquid cooled equipment. And I think thats going to be here sooner than people
think. When we roll it in and start seeing what we can get with that gas pedal,
thats when I think the lights are going to turn on for people. Like, Wow, I today
build infrastructure with DR that goes no more than 50 percent of my capacity.
Thats really wasteful. Im only using half of what I can out of these servers yet
Im still paying the wattage penalty. Imagine if you put in half the amount of
equipment and you had the ability to overclock those to 130 percent when you
need them. Youve got the gas pedal when you need it, because its really only four
hours per day for us where its the peak of the peak. The rest of it Im going down
to these low numbers.
What were looking at is both horizontal and vertical scale. So, Ive got 100 racks
of equipment delivered for search. If I need all 100 racks for that peak of the peak
and I overclock them to that point I can probably use 70 racks for it. The vertical
scale: I can take my 70 racks and make them act like 100 racks by overclocking.
172
Then I go back and say Well, when I go down to here I cant go too slow because
The Art of the Data Center
our search function requires a certain amount of latency to deliver a search result
to a customer. So theres the threshold at the bottom you say, Well I would never
go below 2 GHz. Ok. So downclock them all to 2 GHz. Youre still delivering to
the performance of the application. Then, if you only need 30 of the actual racks
take the rest of them out of the pool and put them in hibernate. And as you see
the triggers going up where its starting to increase add another rack in. Thats the
horizontal scale.
But you see what I mean is we build the infrastructure to be able to have these
cabinets scale up and down and horizontally as we need them, because we know
what the IT workload is and we tie back in the infrastructure so that the engine runs
efficiently. Think of a car. People are designing Data Centers without understanding
how the engine inside operates and the people designing the engine are not
expecting or worrying about anything in the cooling or what-not. So they drop an
engine in and theres a disconnect. But when you have them designing the whole car,
because the car itself is the Data Center, the engine is the IT and the control system
is the gas pedal and everything else that takes advantage of ityoure tuning for all
those aspects. Thats what were looking for. Ultimately it becomes one machine with
all components working together and knobs that automatically turn themselves.
Were also looking to converge in the infrastructure and allowing the Data Center
because weve got I think over 240,000 points monitored in that Data Center in
Salt Lake City and that we go into a PI System and that PI System is the stuff that
runs nuclear power plants. It scales to millions and millions of points. We want to
take that data and roll that back into our engine that is the application so they can
start to understand, Oh, its currently nighttime in Phoenix and I can overclock all
my systems to here and ship the workload over there, because now its even colder at
night so I can get even more performance out of those chips because its delivering
70 degrees (21.1 Celsius) instead of 87 degrees (30.6 Celsius) during the heat of the
day. But the whole engine is tuned and the applications now can take advantage of
that. The whole stack is connected. Thats where our ultimate goal is, is to be able
to have a finely tuned machine that only consumes the wattage it needs but can
perform based on whatever business demand comes.
9: eBay
Data Center design principles. I have taken away some from our
conversation already: modularity, scalability, look at the Data Center
from end-to-end, expect liquid cooling in the future. Is there any other
advice that you would offer someone?
Multi-tier. If you build a modular infrastructure you should be able to deliver
whatever tiers youre looking for. If you build out Tier IV, youre going to have
Tier IV. If you build out a center that can scale to Tier IV or you build out a
section thats Tier IV and the rest of them can scale between Tier I and III, youre
going to give yourself a lot more flexibility. The majority of companies out there
still have at least half or less of their equipment that really requires Tier IV.
Even the banking industry, certain transactional things and all of that its the data
where its stored. Its the network that has to be in the Tier IV, but the compute
engine thats doing that theyve got to point it at multiple locations, the same
thing as our search. You should be able to put that in multiple locations at a lower
tier. So, right sizing your applicationstheres another one.
I guess the other piece is still the same mantra: can the Facilities and IT teams
work together? Are they approaching the design with a unified front? Are they
looking at the problem together?
Ive got a certain amount of capacity inside of our Data Center in Salt Lake that
The Art of the Data Center
because today Im filling up the space, I would have consumed all my power
already with compute equipment unless I now am building a center that I can
pull out certain things that are lower tier. So, the capital investment of a Tier IV
what you really probably need to do is how much would I actually build Tier IV,
and I still would be really surprised if people would build more than 50 percent.
If I am building 50 percent Tier IV and 50 percent Tier II I can build two to
three times the amount of Tier II and give myself a lot more runway, up to four
times the amount that I would within Tier IV. If I build that out modularly I
should be able to now scale it up as I need. Were looking at 75 to 80 percent of
our stuff is going to be removed out of our Tier IV. Thats a big deal.
Talk about runway. I gave myself another 5, 10 years because Ive just gotten rid
of the majority of load in there. And Im only putting the core network and the
large storage devices and certain databases and other things that really cant go
down, in there. And those scale a lot slower than everything else. But the ones
that scale are the search nodes, are the front end nodes, the ones that we want
to replace every two years because thats where all the workload is and they call
the databases, they use the network. But all those computers are in the front,
consuming a ton of power.
Again, its the multi-tenant environment. If you have a generic footprint what
we found is the smallest unit is going to be a rack. When we roll in a rack of
whatever its going to be allocated to, a business unit or a function. So if weve got
generic cabling infrastructure, power infrastructure, etc., to be able to roll that in,
then we dont really care. We want to have secure environments, we can still lock
the cabinets, we can still put them in different sections but its still anything can
go anywhere at any density and attached to any project.
That requires the network team to figure out how they would isolate. That
requires the IT teams to figure out how they would distribute for disaster recovery
for failure domains inside of the equipment. Theyve got to think through it. But
weve given them a flexible environment. Today in a lot of the centers I have to
go figure out which blade slot in which chassis has enough power and enough
connectivity to be able to add capacity. Thats wrong. What I should be able to
do is say, What capacity do you need in IT? and I can accommodate it wherever
I am. I can roll that into the Data Center and plug it in and go. That agility and
that speed, that time to market, is whats really, really important to us.
That flexibility in a traditional enterprise IT organization makes it simple. You
acquire companies, you downsize, you do a tech refreshall those different
activities you still have a flexible infrastructure that allows you to accommodate
175
them. You may not have a high growth business such as an Internet based
9: eBay
company as eBay or Amazon or Google and those things but you definitely have
the same philosophy about modularity, enabling flexibility inside of a center
to grow at your pace. And to do it very, very cost effectively both in capital
investments as well as efficiency of operating them.
Ill do that on everything all the time. I wont get to the single power cord. Youve
The Art of the Data Center
got a lot of people out there that are going that way, too. Low wattage CPUs,
low frequency, cheap commodity hardware with single power cords. They will go
down. As long as you have everything that they can go down, ok. But if I dont
have to go down why should I go down?
I am actually adding an extra power supply and those things in there, but Im not
adding all the UPS and gen and all the other stuff on the other side that cost a
lot of money so Im still getting all the major benefits out of it. The net-net is I
have two power sources for every device in the Data Center and by the way we
added additional UPS feeds for the things that really required more runtime like
the network infrastructure thats inside of the building. Its Tier III because it still
goes through the Tier IV backbone but I dont want those to go down. But thats
like less than 3 percent of the load and the space in the Data Center. The rest of
it, its all two-cord, UPS gen on one, street power on the other. So that was a great
learning for us.
9: eBay
Figure 9-5 Half of the Data Centers capacity is provided by way of containers on the roof.
The other lesson learned in this is if you dont have to go that fast to build it,
dont. If you can give yourself more time, do it. You do make compromises. This
team has been really good, they havent compromised on any of the major pieces
we needed, but schedule did limit the amount of vendors who could participate.
We couldnt use bundled or coupled gen-UPS units, because we couldnt get them
in time or their size was a bit too big. So that limited some of the vendors from
being able to be in it. Then we had to go back and now have decoupled gens and
UPSs. Its working but I would much rather have those modular units because
they work better when theyre coupled together. The same manufacturer is
responsible for that handoff between those units. Its just a finely tuned machine
versus two independent vendors hoping that theyre going to lock the bus and be
able to switch.
Oh, and another lesson learned is leading power factor. Not a lot of companies
are turning on a half a megawatt to a megawatt at a time of load. If you dont have
an inline UPS thats adjusting the power factor so that the gens dont see any issue
178
you have to figure out how to manage that. If the leading power factor is too high
The Art of the Data Center
on the server equipment it will shut off your gens. The big playersyou look at
Microsoft, Google, uswhen were putting up these containers of 1,000 to 2,000
servers at a time you have to really consider that. So we had to put some stuff in
the middle to correct that, to filter that to allow us to be within the tolerances of
leading power factor. And thats primarily because of the way the design is, I dont
have a double conversion UPS because that gets rid of the leading power factor.
You mentioned the Data Centers 14,000 sq. ft. (1,300.6 sq. m)
hosting area. How much additional area is used for electrical and
mechanical systems?
Its close to two times the floor space. Because we used larger units, weve got 3
MW gens so were adding those chunks in three at a time and the UPSs are 2
MW. There was a kind of a mismatch in those sizes but we know that well be
adding the next brick as we need it. Because we still have N+1 in the gens. So, 2x
I think.
The way the design was put together is, weve got a 55 degree loop because we
still have to have chilled water to some things. So, 55 (12.8 Celsius) and then the
other one is the 87 degree (30.6 Celsius) loop, so its a condenser loop. And both
of them are directly accessible to the equipment itself. We can choose over time
which one were going to build out. So if weve got 12 MW of capacity that we
need to deliver and then the N+1 in that I can now say 8 MW of that is going to
be condenser so scale those up. Or if we realize that were getting more efficiency
of some of the chilled water you can scale that up too. Or we can scale the chilled
water and then scale it back and scale it up with the condenser over time. Because
not everything is going to fit on that 87 (30.6 Celsius) loop and we cant mandate
that right now. We still need the flexibility.
179
If I look at the grand scheme of it, its less than 5 percent (of our capital cost)
9: eBay
that we put in there to be able to have that flexibility. Again, having it from
the beginning, that the design is flexible and all of the design engineers are
understanding the problembecause we spent a lot of time with them explaining
the business challenge and then what the ultimate numbers are going to be.
Finally we came down to a density per rack footprint that we needed to have
based on the space and based on the available MEP yard space and that that we
could say this is the sweet spot and then well scale it up over time.
The ground floor, we knew we would have stuff coming in at less than 10 kW in
some areas. But other ones were going to have rack and roll thats coming in at
25 or 28. Ok, well we can handle both and then we just scale them up over time
by adding in modular units. As we get denser we add more modular units right
on the floor. They take up space but as we replace those and potentially roll in
the liquid solutions we gain all that space back again. So, the entire ground floor
I can have 70 percent of it at 40 kW a cabinet and thatll work. Thats a big deal,
hundreds and hundreds of cabinets.
With all of the steel you had to put in place, what is the weight
bearing capability of the Data Center building?
We can handle containers with all the supporting components at 125,000
to 140,000 lbs. (56.7 to 63.5 mt.). We can handle anywhere from 10 to 12
containers on the roof.
We have one vendor thats coming in at 98,000 (44.5 mt.), another one that came
in at 112,000 (50.8 mt.). The next ones when they come in we dont know if
theyre going to be heavier or less if theyre all liquid cooled because liquid would
have more weight but we would have less infrastructure as well. We dont need
all those air movers. We need liquid infrastructure to the chips and thats low
flow liquid. So its that same amount of liquid going to the container its now just
distributed directly to the devices instead of distributed to the air handlers. And
then the other one thats using all outside air cooling, the swamp cooler stuff, that
just is city water and not a lot of it.
I love this because the Data Center has 100 percent outside air cooled solution.
Right next to it is a design thats using a heat exchanger with a primary source is
the 87 degree (30.6 Celsius) loop and if we happen to go over 87 (30.6 Celsius)
because of some fluke over time it exchanges with the 55 (12.8 Celsius) loop to
cool down the 87 (30.6 Celsius) to 87 (30.6 Celsius). And everythings tuned.
180
It was a really cheap, innovative way for us to go back and get direct hot water
The Art of the Data Center
9: eBay
Figure 9-6 A computer circuit pattern adorns one of the Data Center entrances.
182
The Art of the Data Center
Figure 9-8 Sliding doors at the end of rows 7 and 8 open (left) and closed (right).
183
9: eBay
Figure 9-9 Containers are pre-populated with more than 1,500 servers each and then placed on the
roof by a crane.
Essential Details
ORGANIZATION: Facebook
Location: Prineville, Oregon
Online: May 2011
Notable features: 277 volt power distribution. Evaporative cooling system. Power over Ethernet (PoE)
LED lighting. Custom servers. LEED-Gold certified.
Time to design and build: 15 months for 147,000 sq ft. (13,656.7 sq. m) first phase
Size: 307,000 sq. ft. (28,521.2 sq. m) total, with about 261,000 sq. ft. (24,247.7 sq. m) of hosting
area and 46,000 sq. ft. (4,273.5 sq. m) of electrical room space. Mechanical systems are located in an
enclosed area on roof and not included in size figures.
Power: 30 MW available for IT
Tier: III
Cabinet locations: Undisclosed
Infrastructure Delivery: Power, cabling, and cooling are all provided overhead
Structural loading: 250 lb. per sq. ft. (1,220.6 kg per sq. m)
Fire suppression system: Pre-action dry pipe system with VESDA monitoring
Imagine having to create your companys first
Background
The Interview
10: Facebook
Your company has rather famously gone public with information about
its Data Center in Oregon, through the Open Compute Project. At what
point in the project was it decided that you would be sharing your
Data Center design information with the world?
We started this design back in early 2009 and developed it by end of 2009. It
really came from our fearless leader, (Mark) Zuckerberg. He said, This is not
our core business, Data Center design, and if we can share with the world and
everybody can save energy... He had this vision and he wanted to open it up.
The second thing is that as we developed this new power supply running at 277
volt and the localized DC UPS system. You know, if more people can buy this
stuff quite honestly we can drop the price as well. The power supply is really the
commodity, so the more people buy it the price will come down.
So those are really the two main drivers from the company perspective.
LEED Certification
Leadership in Energy and Environmental Design (LEED) is a rating system for the design, construction, and
operation of green buildings. Facilities can achieve ratings of certified, silver, gold, or platinum based on
criteria within six categories: sustainable sites, water efficiency, energy and atmosphere, materials and
resources, indoor environmental quality, and innovation and design.
188
Did knowing that your Data Center information was going to be public
The Art of the Data Center
influence the design? You were not only trying to design an effective
Data Center for Facebook, you were trying to design something that
was intended to be a model for others to emulate.
That is correct. If you look at the design, in all these years we knew that if
we increased the input power voltage we will be able to gain some additional
efficiency in these power supplies but industry in general they didnt want to
develop a new power supply running at 277 volt that comes right off from
the 480 volt system. They will continue to use this 208 volt system that
requires additional transformation from 480 volt, perhaps to an inline UPS
(uninterruptible power supply) system, things like that.
So, yeah, youre absolutely correct.
10: Facebook
Figure 10-2 Facebooks custom servers feature no paint, logos, stickers, or front panel, eliminating
weight and materials.
The second thing is that the UPS system that we designed does not have all of
The Art of the Data Center
the components like an inline system, like inline rectifier modules, you dont have
an inverter, you dont have static bypass switches, you dont have maintenance
bypass switches, and all that. The design is just the 48 volt DC battery banks in
parallel, with the one common bus. Thats basically what it is. I called it as a UPS
system because its an uninterruptible power supply but it does not have all of the
components like an inline UPS system.
And, note that this is an offline system. Its not an inline system at all. During the
normal operation the DC UPS system is basically all in standby mode.
How did the design for the enclosed mechanical area above the IT
hosting space come about?
Originally we were going to install rooftop units, on top of our roof space, above
the IT space. What happened was, when we started laying out all of the rooftop
units, it was going to take a lot of roof space and we were running out of roof
space. In addition, on cold days in the winter, maintaining this equipment up
there it would be a very difficult thing to do.
So what we did is instead of buying packaged rooftop units we just built the
penthouse wall around it and put all of the inside equipment in the penthouse.
Its not a people space, you can call it a single giant rooftop unit if you want to.
You mentioned the misting system. Can you walk me through how
10: Facebook
thats used and how the evaporative cooling system works?
Yeah. Lets start with the city water. We kill the bacteria, we eliminate all these
substances. Then we pump that water up to what we call the pump skid. This
pump will create the water pressure. The nozzle sprays in very, very fine water
mist, which creates the misting system. Then basically the fan array will pull the
air from outside. The system is horizontal, in parallel with the air stream, and it
cools that air. Then it dumps it over into the Data Center.
The pump skid is right now configured as a 2N system. We got two pumps with
manifolds and we come up with a common manifold and from that point on we
have many misting zone lines. If you see the picture you will probably see it, the
misting system you see a bunch of pipe lines, like a spiderweb. Each one or two
lines are a zone. And then if you dont need to turn on all the misting system, like
out of maybe one through seven zones, only zone one would be turned on and
zones two to seven would be turned off, with everything in-between, because the
skids are controlled on VFDs (variable frequency drives).
Figure 10-3
When needed,
warm server
exhaust is added
to incoming air in
a mixing room.
192
Figure 10-4
The Art of the Data Center
You have the evaporative system in place and also an air economizer.
How many days per year do you anticipate being able to use outside
air for cooling?
Throughout the entire year were going to be using outside air. This is it. The
outside air and we just cool it with the misting system and dump into the Data
Center. One hundred percent of the time.
During the wintertime, since bringing in 100 percent outside air would be
too cold for the servers, we recirculate hot return air from servers to bring the
temperature up to an acceptable range. So the outside air damper during the
wintertime is only partially open and mixes the temperature up to a warmer
setpoint.
193
10: Facebook
Were using the ASHRAE (American Society of Heating, Refrigerating, and Air
Conditioning Engineers) TC9.9 standard 80.6 degrees Fahrenheit (27 Celsius)
as the maximum inlet temperature and 65 percent RH (relative humidity) as the
maximum RH.
How long did it take to design and build the first phase of the
Data Center?
The design and construction was one year. Well, a little bit more than one year
because we kind of cheatedwe actually started our design work a little bit ahead
of time. We were doing a little R&D work here and there so we were able to use a
lot of that design. I would say about 15 months from design to completion of this
project.
Even at 15 months that still seems fast for a facility of this size that
has a lot of innovative technologies and youre making your own
custom server design as part of the project. As you said at the start of
this interview Data Center isnt necessarily core to your business. For a
company that doesnt have Data Centers at the core you seem to have
done a pretty good job with it.
Well, let me back up. Our site acquisition stuff like that I did not count that time.
Im talking about the Data Center design and construction, probably about 15
months. Because we were doing this R&D site at the time we were doing a lot of
engineering work up front. We were able to use a lot of that.
194
Lets say, if you have to build a Data Center and were giving you all this Data
The Art of the Data Center
Beyond the technical details that are included in the Open Compute
Project documents, are there any particular Data Center design
principles that you followed with the project?
When I joined two and a half years ago, there was no Data Center design team. I
was it. I came from the Data Center design industry, construction industry, and
then we started pulling a bunch of the experienced people on board. So, we knew
exactly what we were doing. Its not like we were spinning the wheel or trying
new things. That was not the case.
Certainly. Is there any general advice that you would suggest someone
keep in mind as they embark on a Data Center project?
The only recommendation I would like to give industry is dont believe
everything the server manufacturers say. Their nameplate data, obviously they
are very conservative numbers. As a user I think they should really do a little
more research on servers and understand the boundary of the server operating
parameters. And then dont be afraid to push that boundary. I think thats the key
to have a successful project.
During your project were there any surprises that came up along the way?
10: Facebook
Yeah. Because when I came from another industry we built the Data Center but
we never really built the servers. So when we started developing the servers and
actually the servers they can take a much harder, harsh environment. You can push
the humidity level up to 90 percent. It will work, no problem. Or the low side
humidity condition. People talk about static discharge, its going to damage the
memory chip, CPU, blah, blah, blah. But the truth of the matter is that every rack
has been grounded. With the proper procedure, when you touch the servers you
can ground yourself. Its basic process and procedure to touch or remove servers.
You could do all that and you dont need to really worry about low-side humidity
either. There are a lot of things you can do. Those are the kind of things I found
out. I think the industry in general theyre too conservative.
I was quite surprised actually how much we can push the boundary of server
operating conditions. We didnt quite do that but next phase, next project were
actually pushing the boundary a little bit harder.
Figure 10-5 Power, structured cabling, and cooling are all provided overhead in Facebooks
Data Center.
196
Your Data Center is obviously very new but if you go back and design it
The Art of the Data Center
I read about your Power over Ethernet (PoE) LED lighting system.
What prompted you to use that particular solution?
First of all, as you know the LED light fixture lasts a really long time. Another
reason we wanted to use that system is we have hot aisle containment doors, we
have a ton of doors. Lets say somebody opens this door and doesnt shut it tight
and leaves it open. All the hot air will be mixed with the cold air and we will
not know about it. We could have put a door contact in and send it over to the
Building Management System but even though we do that sometimes its hard to
spot these doors. So by using these LED fixtures you can actually integrate that
with a door contact. If somebody leaves that door open the whole bay light we
can make it flash.
So, there are a lot of controls. The functions are built into that system. And
this is why we chose that system. And plus its obviously energy savings and
its incredibly low power consumption. But quite honestly the ROI (return on
investment) wasnt quite there. The truth of the matter is its an expensive system
and it wasnt there. We dont have to change the fixture every two or three years,
itll last almost 15 years.
If you leave the lights on all the time it might be a different story but we have
what we call a lights out environment. If nobodys there we turn the lights off.
When we ran that calculation under the lights out environment the ROI was not
there. But the control system was very interesting and we were able to use that
system to monitor the doors.
197
For your fire detection and suppression system you have VESDA
10: Facebook
monitoring and pre-action sprinklers. Is that a dry pipe system?
Yeah, it is a dry type system. As you know, if you leave that dry type system
and you leave the pipe empty over a period of time youre going to see some
condensation building inside of this pipe. If the pipe is perfectly sloped and if you
have the drainage, thats not an issue. But I can guarantee you when these guys
installed this piping its never going to be perfectly sloped. So what happens is this
condensation happens and youre starting to collect this water and it will collect in
the low point. And then it will start corroding and you will have a little pinhole
and then water will be starting to drip.
I experienced this problem with my previous employer and Data Center.
Changing this piping in a live Data Center was a huge challenge. And so what
we did here is that, not only is it dry type but we used nitrogen gas rather than
compressed air. When you fill it with the nitrogen you dont have the oxygen
so the corroding problem goes away. If you look at the National Fire Protection
Association (NFPA) website they talk about this corrosion problem a lot and
thats exactly what they recommend, too.
Figure 10-8
10: Facebook
Air passes through mist eliminators, at
left, and propelled by variable speed fans,
at right.
Essential Details
ORGANIZATION: Green House Data
Location: Cheyenne, Wyoming
Online: January 2008
Notable features: All power for Data Center comes from or is offset by locally generated wind.
Virtualization, high-efficiency air conditioning, airside economizer, hot aisle enclosures.
Time to design and build: 12 months
Size: 9,500 sq. ft. (882.6 sq. m) total, with about 7,500 sq. ft. (696.8 sq. m) of hosting space
Power: 1 MW overall; IT usable capacity of 850 kW
Tier: III
Cabinet locations: 200
Power Density: 4.25 kW average per cabinet, 12.5 kW maximum
Infrastructure Delivery: All infrastructure is delivered overhead
Structural loading: Undetermined
Fire suppression system: Novec 1230
Wind isnt known for blowing in straight
Background
The Interview
Lets start with some background about your facility. What prompted
you to build your Data Center in Wyoming?
In 2007 a good friend of mine and I were sitting down over coffee in Jackson
Hole, Wyoming, talking about how there were no Data Centers in Wyoming. We
began researching and determined that Wyoming could definitely accommodate
a high density, multi-tenant Data Center but it would need to be a nationally
unique business model. This is where I had the idea of tapping into Wyomings
wind resources. This direction then lead me to research methods for Data Center
efficiency for not only sustainability but also cost efficiency.
203
How long did it take to design and build the Data Center?
We built our facility in a highly modular fashion. With an unknown amount of
demand and the desire to be an entrepreneur and build it and they will come, we
went down the path of building it out in a modular fashion. We have a 9,500 sq.
ft. building (882.6 sq. m) and we built out the first private suite of 1,000 sq. ft.
(92.9 sq. m) to start and then have just grown modularly since then. Now were
launching another 3,000 sq. ft. (278.7 sq. m) of our facility as a suite.
Your facility doesnt have a raised floor. Did you forgo it for any
particular reason?
We were retrofitting an existing building and the ceiling heights werent high
enough to allow us to use a raised floor. We were able to come up with a design in
our building specifically that is relatively unique just because of our floor layout
that allows us to use overhead supply and hot aisle overhead return as well.
So, we are committed to getting our wind energy a portion of it through the
The Art of the Data Center
local power company and then offsetting it and covering it 100 percent through
renewable, Green-e tags. Thats one component of it from the green perspective.
Probably more important isand what we tell our customersthe greenest
electron is the one you didnt use. We operate at probably about 90 percent
greater cooling efficiency than most Data Centers. And were doing that by
leveraging the outside ambient air in Wyoming to cool the facility almost 365
days a year.
How can someone who wants to power their Data Center with
renewable energy, but for whom on-site power generation is
impractical, do so? Whats the process for making that happen instead
of just using the standard electrical supply from the local utility?
We talked to our local power company about how we can leverage their
investment in wind energy. So it all starts with the local power company, working
with them to figure out what type of accounting capabilities do they have to
account for the wind energy production. Then what weve done to purchase our
own renewable wind energy credits is gone through a company called Renewable
Choice in Boulder (Colorado) that then actually purchases the wind back from
other wind farms in Wyoming and specifically ones right near us that were not
able to tap into.
206
area is cost?
Yeah, absolutely. Its cost-related. Really, when you boil it all down, its cost per
kilowatt hour related. For me to produce it here on-site at my facility its cost.
For example, 40 cents per kilowatt hour, which is really expensive power. Where,
literally across town, at utility scale they produce it at probably 4 cents per
kilowatt hour.
Your website mentions aligning hot and cold aisles and redefining the
cage space concept. Can you elaborate on that?
In essence, there are zero cages in our Data Center facility and if we were to
put a cage for security purposes it would still fit within our hot aisle, cold aisle
containment scenario.
207
The problem with cage spaces is what we call the chaos factor. When you put
Figure 11-3
Plastic enclosures
above the cabinets
isolate hot and
cold airflow while
door panels control
physical access.
208
Youre using a clear plastic enclosure at the top of your cabinets that
The Art of the Data Center
Were there any surprises during the design and construction of the
facility?
Yeah. The biggest surprisehistorically and not so much moving forwardwas
the lack of Data Center engineers willingness to build a non-traditional Data
Center. It was definitely us pushing them to think outside the box to build this
the way we wanted to build.
Do you think the various technologies in your Data Center are universal
The Art of the Data Center
Your Data Center is not very old but if you could go back and start over
again, what if anything would you do differently?
The interesting thing is we do get to start over, and we are starting over with
this next 3,000 sq. ft. (278.7 sq. m) suite. What we are doing differently is that
we wont have a full DX backup moving forward. We are comfortable with our
ability to be cooled by 100 percent outside air and so are our customers. This is
similar to Facebooks Open Compute Project, which is cooled by 100 percent
outside air and yet still falls within the ASHRAE standard.
Any advice that you would give or design principles you would offer to
someone preparing to start a Data Center project?
The biggest piece of advice that I would give is find engineers that agree with your
principles, so that you dont have to battle the engineers to create the Data Center
youre trying to create.
Figure 11-4
Power is
distributed
overhead to Green
House Datas Data
Center cabinet
locations.
212
The Art of the Data Center
Figure 11-5 A closeup of the electrical bypass switch for Green House Datas server environment.
213
Essential Details
ORGANIZATION: IBM
Location: Research Triangle Park, North Carolina
Online: November 2009
Notable features: Modular design, extensive sensor network that monitors and manages electrical and
mechanical system efficiency, use of both air- and liquid-based cooling. Rainwater harvesting. Ninety-five
percent reuse of pre-existing building shell. LEED-Gold Certified.
Time to design and build: 19 months
Size: 160,000 sq. ft. (14,864.5 sq. m) total, with 100,000 sq. ft. (9,290.3 sq. m) of hosting space.
Power: Two 21 MW power feeds, 15 MW for IT
Tier: Declined to estimate
Cabinet locations: 960
Infrastructure Delivery: Electrical infrastructure, structured cabling, and cooling are delivered under a
36 in. (91.4 cm) raised floor
Structural loading: Undetermined
Fire suppression system: Wet pipe system, with VESDA detection
When it comes to battles of man versus
Background
The Interview
What functions does this Data Center in RTP serve for IBM?
It serves two major functions. IBM provides a lot of outsourcing content for
customers and its an outsourcing source of our Data Centers. We have roughly
8 million sq. ft. (743,224.3 sq. m) of Data Center space in the outsourcing
segment that we manage for our customers on their behalf. This is our second
largest Data Center.
The second major purpose for this Data Center is our public cloud computing
offerings, of which at the moment there are two offerings that are offered out of
that Data Center.
217
12: IBM
something new from the ground up?
There were basically three things that we were looking at, which were cost,
schedule, and resources. In this particular case we were renovating old warehouse
space and manufacturing space, which had the power characteristics already there
at the site and also the hardened raised floor and a dual-story roof. That and the
fact that we had been a part of the Research Triangle Park for quite some time.
As a result of those items a couple of things came into play. One, the schedule
was shorter because we didnt have to go buy land or build a building. Two, the
building was fully depreciated so the cost of the project was less. And three, the
schedule in general came in because of the lack of needing to add things like
power or hardened raised floor, that kind of stuff. Other characteristics of using
an existing building made it very lucrative from the standpoint of schedule, cost,
and labor.
LEED Certification
Leadership in Energy and Environmental Design (LEED) is a rating system for the design, construction, and
operation of green buildings. Facilities can achieve ratings of certified, silver, gold, or platinum based on
criteria within six categories: sustainable sites, water efficiency, energy and atmosphere, materials and
resources, indoor environmental quality, and innovation and design.
218
All that work had to be done up front from the standpoint of determining how
The Art of the Data Center
the project was going to be implemented in order to do those things during the
project.
How does modularizing the Data Centers design help save time and
scale capacity? I assume this is more than just building out a few rows
at a time.
In this particular implementation we had 3 MW modules and we had room for
five of them that we designed in, and the cooling that matches the power because
the power turns to heat dissipation. So we created a 3 MW module that had the
UPS (uninterruptible power supply), the generators, the chillers, and then we
decided where those would goeither inside in an MEP (mechanical, electrical,
and plumbing) room or outside as far as generatorsand then we reserved the
space for them but we did not install them. So right now today we have all the
generators for the first 6 MW and we know in the existing 60,000 sq. ft. (5,574.2
sq. m) where the next set of generators are for the next 3 MW. We also had to
219
pre-install some of the cable trays and make the water pipes big enough to be able
12: IBM
to hold the eventual load, but the amount of additional cost to do that while you
were doing the build was negligible compared to having to go back and add that
after the fact.
The modular approach helps you bring in the 18- and 24-month projects down
to under a year and it makes them smaller projects. Instead of building a 15 MW
Data Center were building five 3 MW Data Centers.
Figure 12-2
Thermal storage
tanks for IBMs
Research Triangle
Park Data Center.
220
The Data Center had first 6 MW and later 15 MW of capacity for IT.
The Art of the Data Center
What are the overall power feeds into the Data Center?
You figure it was designed with a designed criteria of a PUE of 1.4, so you can do
the math on that.
The benefit there is we did some things that improve the reliability, decreased
the risk, but decreased the cost. Ill give you an example of that in the cooling
distribution system. Typically a cooling distribution system would have, even
with a waterside or airside economizer, the chiller would be in line with the
economizer which would be in line with the pumps which would be in line to the
CRAC (computer room air conditioner) units out on the floor. So, there would
be a kind of a one-to-one mapping of those.
What we did we criss-crossed the connectivity of those major components so that
any chiller can supply any flat plate heat exchanger which can supply using any of
the pumps to any of the CRAC units. And those were just water valves. Thats not
a very expensive thing to get that extra resiliency.
I can see how that would give you some additional redundancy within
the system. Does that also increase the amount of power density or
heat load that you can support at a given cabinet location?
No, thats predominantly just a resiliency statement.
One of the other things that we did was, traditionally you would size the chillers
and put chillers on the UPS units so therefore increasing the amount of UPS that
you needed in addition to the UPS that you needed for IT. What we ended up
doing was running the numbers economically and deciding instead of putting the
chillers on UPS we just put them on generators. And what we did was we built
three 50,000 gallon (189,270.6 liter) chilled water tanks and we run the cooling
distribution through the chilled water tanks similar to how we put UPS batteries
in line, to use the cooling tanks to condition the water from a temperature
standpoint. Those tanks have 30 minutes of ride-through time, which is more
than sufficient to get the generators up and running and get the chillers running
on the generators at a much lower cost point. That affected operational cost and
capital cost but at the same time improving resiliency and reducing risk.
221
12: IBM
cabinet to cabinet.
Traditionally people have used the term watts per square foot. And watts
per square foot is getting to be the worst metric of a Data Center because it
implies that youve got uniform distribution of power and therefore cooling to
a particular segment of the raised floor. And we have moved away from that.
Instead of having, like, all 4 kW racks or 2 kW racks we wait until we actually
need to run the power and we supply a certain amount of forced air cooling. But
if we need to be able to cool past that well do something like add a rear door heat
exchanger to have the supplemental cooling. So weve disconnected the amount
of cooling you have to have for the amount of power. Also, we can run different
capacity racks.
We did an analysis of how many of each type of rack in terms of power density
our customers were currently using and then we extrapolated to say, Ok during
this first revolution while they are filling up the Data Center how much power
density will they have to have by rack? So we support not only the low density
racks but also support the 8, 16, and now 32 kW racks and we dont strand that
power because we dont run it until the rack needs it.
This Data Center was planned from the beginning to support cloud
The Art of the Data Center
It seems that you have a lot of flexibility with the Data Center.
In addition to the modular approach you can adjust how much
infrastructure to provide at a given cabinet location based on what the
demand is going to be.
Yeah. And the reason we have to do that is, and weve been extremely successful,
were an outsourcing group so we dont know which accounts are going to
sign when and we dont know which accounts are going to have mergers or
223
12: IBM
to make mistakes and we dont want to put ourselves at risk, so we do that by
creating this module flexibility.
Figure 12-3
Pre-piping under
IBMs raised floor
supports future
water cooling in
the Data Center.
Tell me about your rear door heat exchangers and what functionality
they provide.
Liquid cooling techniques are coming back to the Data Center. If you look at
our Z10s, which are our mainframes, theyve got built in coolers that you just
dont see the external water connections. Old mainframes, circa 10, 20 years ago
had water connections to them. So were starting to see liquid cooling techniques
come back into the Data Center and some people who have them are concerned
about that risk. Were not because weve got a long history of using water in the
Data Center not to mention the water thats in the sprinklers right above the
equipment. But be that as it may we want to be efficient, we want to be cost
effective. So if you look at the U.S. EPA (Environmental Protection Agency),
which commissioned Lawrence Berkeley National Labs to go look at what
the latest liquid cooling techniques are that come closer to the Data Center
obviously CRAC units are supplied by water but theyre at the perimeter of the
Data Center, Im talking about within the rackthen you would look at the
results theyve had to what they call chill-offs.
The rear door heat exchanger is a device that fits on the back of the rack and has
won the chill-off the last two years in a row, of all the liquid cooling techniques
from the different vendors. Lawrence Berkeley National Labs did the independent
analysis and it was the most cost effective technique for cooling per unit of cost.
That device is a low pressure device and it actually contributes cooling to the
room for every rack that its on, because what happens is the fans from the servers
or the IT equipment blow across a radiator. That radiator has cold water flowing
224
through it. The net effect, if you put your hand on it, is that it ends up blowing
The Art of the Data Center
cold air into the Data Center because it overcools the amount of cooling thats in
that particular rack. The other thing is that is low pressure water and its a closed
loop sealed system, so should there be a leak it would not leak that much water
and it would not leak it at high pressure. So, again, low risk for the additional
resiliency.
The other thing is you dont need to put in on every single rack. You can just put
it on the high density racks.
12: IBM
point you start using liquid cooling rather than forced air?
Economically its much better than that. Theres the physics involved and when
do you have to. And thats a function of how youre spreading the forced air
around and are you averaging it by putting a high density rack next to a low
density rack. Even at the 6 MW over 60,000 sq. ft. (5,574.2 sq. m) with 32 kW
racks, we could still air cool them from a physical standpoint. From an economic
standpoint what were finding is at less than probably 10 kW racks it may be
more economically feasible to start using rear door heat exchangers. As that
technology keeps improving its going to go down to 6.5 kW.
This Data Center has more than 30,000 utility and environmental
sensors tied in with various software tools. What sort of data are the
sensors collecting and how are you making use of the information?
To start with, youve got the up-down sensors and monitoring sensors for all the
Facilities and the IT pieces of equipment. So, the CRAC units report information
on up-down and how well theyre doing. The generators, the pumps, pretty
much any piece of electrical gear either on the Facilities side or on the IT side is
reporting things. The generators report oil pressure or amount of gas left in the
tank. So thats where theses tens of thousands of sensors are starting to report
226
data and would create an information overload if all you did was look at the data.
The Art of the Data Center
So from that you have to start having whats now called a class of Data Center
information management or DCIM (Data Center Infrastructure Management)
software that starts paring that down to taking the data and making it
information and knowledge that you can use.
The other thing thats being reported from the IT equipment is temperature
data and energy use data, how much energy a particular server is using, because
the new Energy Star rating for servers requires that the energy fluctuate as the
utilization fluctuates which means youll have much more stranded power than
you did before because usually when you turned a machine on it didnt vary much
depending upon what utilization it was running.
Figure 12-4 Use of waterside economization provides free cooling about 3,900 hours per year in
IBMs Research Triangle Park Data Center.
227
IBM has airflow modeling technology associated with this Data Center
12: IBM
as well. Is that accomplished through these sensors?
Thats something thats a little bit different but integrated with our overall DCIM
strategy.
IBM prides itself on having a very large investment in R&D, to the tune of $6
billion a year. A lot of that goes to our IBM Research organization and our IBM
Research organization came out with this MMT (Measurement and Management
Technology). Originally it was a mobile technology that had sensors at different
heights and could build you a CFD model of the Data Center to identify hot
spots. Its a temperature model and could model airflow as well. Well that was a
snapshot and you would have to move the portable technology around each time
you wanted a snapshot. What weve done now with the Raleigh Data Center is
add permanently mounted sensors above and below the floor at different points in
the rack and at different points in the rows and the aisles. And we can generate a
continuous CFD model or snapshot dynamically.
Once you have that information you can tie that into hot spot analysis in real
time or you can plug that into your decision making process on where you place
the next piece of IT equipment in relation to the IT equipment you already have.
So you in other words put it in the cold areas not the hot areas. And then you can
also use that information for capacity planning for where do I need to put the
next cooling technology or when were doing analytics, whats the optimal cooling
solution for the entire raised floor and should I eventually move workload around
the Data Center to even out the cooling and mitigate hot spots?
not be there or the cable cutouts arent filled with cushions or the perforated tiles
The Art of the Data Center
are in the wrong place. It helps us do that type of analysis. Likewise it helps us to
identify most older Data Centers, especially ones that had liquid cooled or a lot of
hot equipment, there are way more CRAC units in a facility for redundancy than
need be. Its actually also advised us on turning off some of the CRAC units and
saving that energy and the life of that CRAC unit.
This Data Center has a raised floor. What infrastructure are you
delivering overhead and what are you delivering by way of that raised
floor?
Overhead were running the water for the fire suppression system. Underneath
were doing the forced air. Were also doing the electrical and cabling underneath
as well. We looked at overhead cabling. Our criteria for Data Centers has been
better service at lower risk and lower cost. When all other things are being equal
as far as same amount of risk, same amount of cost, we ask the hardware planners
and the people who are working in that space what they prefer and if there wasnt
a cost differential we went with what they preferred. They much rather prefer
under the floor cabling because pulling cables under the floor is easier and quicker
than moving a ladder around upstairs. And they were approximately the same
cost.
The return air plenum is overhead. We have perforated tiles in the cold aisle on
the floor and perforated ceiling tiles in the hot aisle, with a false ceiling therefore
bringing the return air back to the CRAC units.
Its a 36-in. (91.4 cm) raised floor. We found raised floor being still the most
economical way to cool the base set of equipment in the Data Center because the
customers arent investing yet in higher density equipment.
What sort of fire detection and suppression system does this facility
have?
Theres always been a raging debate about charged dry versus wet. IBM prefers
wet. The insurance companies in the area dont care which one has a bias. We also
use flexible piping where the head of the sprinkler is so that we can reposition
it over the IT equipment. The other thing is the sprinklers themselves have
229
individual trips so that if one trips it just trips over that equipment not over a
12: IBM
section or an aisle of the rows of the raised floor. Weve got VESDA early smoke
detection systems as well installed there.
When you think about how this Data Center has been designed, is it
a universal design that can be done by anyone for any facility or are
certain technologies only appropriate based on the mission of this
Data Center or where its located?
There are a couple of things that are geographic specific, like the economic value
that you get from a flat plate heat exchanger. In other words, we have a facility in
Boulder that will use it more than the facility in RTP. The other thing we looked
at was things like CHP or combined heat and power to have a gas turbine.
Youve got to understand that some of those are economically driven not
technology driven. If youve got a power source like a CHP system, natural
gas, that is 8 cents a kilowatt hour that would be beneficial in the northeast
where youre paying 13 cents per kilowatt hour but wouldnt be beneficial in
Raleigh where were paying 5 to 6 cents a kilowatt hour. The technologies have
an economic component which cannot be ignored. So although some of the
things are proven technology theyre not economically feasible for that geography
compared to other technologies.
So, yes you use a modular design. Yes it can go anywhere. The piece that some
people cant do that we did, which we leveraged our Global Technology Services,
Site and Facilities organization (the IBM group that provides Data Center
consulting to external customers), was they did the analysis of what would a 3
MW module consist of from an electrical and a cooling standpoint. Not only the
low initial cost of capital but the operating cost as well. They went through the
major vendors equipment and provided us that guidance on, this is what will
match your electrical with your cooling and this is the model you need to invest
in because of its operational characteristics, because of its variable speed drive,
its variable speed fans. The variability that has the control content where we can
control that variability so as utilization goes up and down on the floor and the
heat goes up and down on the floor, the cooling can go up and down as well. A
lot of people can duplicate that extensive analysis.
230
Any surprises that came up during the design and build of the
The Art of the Data Center
Data Center?
Yeah. A recommendation that we make to everybody thats building a Data
Center is that when you commission the Data Center that you actually put the
type of load to test everything. So we actually did put a 6 MW heat load out
on the raised floor in the different zones and tested it. What we found was our
chilled water tanks lasted longer than we thought their design point would be, so
we actually have more ride-through time than we thought that we would.
Figure 12-5
An under-floor view
of the piping for
water cooling in
IBMs Research
Triangle Park Data
Center.
231
12: IBM
Figure 12-6 Use of both air- and liquid-based cooling allows the Data Center to support different
cabinet power densities.
Figure 12-7 Electrical switchgear for IBMs Research Triangle Park Data Center.
232
The Art of the Data Center
Figure 12-8 3 MW of wet cell battery capacity provide 15 minutes of ride-through power.
Figure 12-9 Six 2.5 MW generators provide standby power for IBMs Research Triangle Park
Data Center.
233
12: IBM
Figure 12-10 Cooling tower fans for IBMs Research Triangle Park Data Center.
Figure 13-1 Intels pioneering use of enclosed cabinets in its Rio Rancho Data Center supports
36 kW per cabinet with air cooling. Images provided courtesy of Intel.
Chapter 13
Intel
Essential Details
ORGANIZATION: Intel
Location: Rio Rancho, New Mexico
Online: July 2006
Notable features: Converted wafer fabrication facility. First Data Center to employ cabinets with ducted
exhaust to optimize cooling efficiency.
Time to design and build: 20 months
Size: 18,000 sq. ft. (1,672 sq. m) total, with 6,700 sq. ft. (623 sq. m) of hosting space.
Power: Redundant substation feeds; IT usable capacity of 8 MW
Tier: II
Cabinet locations: 268
Power Density: 24 kW average per cabinet, 36 kW maximum
Infrastructure Delivery: All physical infrastructure delivered overhead; copper and fiber in separate
cable trays and electrical wiring in dual redundant power busbars. No raised floor.
Cooling solution: Air cooling, featuring enclosed cabinets with exhaust chimneys and air handlers located
in a separate story above the hosting area.
Structural loading: 200 lb. per sq. ft. (976.5 kg per sq. m)
Fire suppression system: Wet pipe sprinkler system with VESDA detection system
Intel Corporation has a reputation in the Data
Background
The Interview
13: Intel
What drove the decision to convert a wafer fabrication facility to a
Data Center?
The evolution of the technology within the semiconductor industry and the
requirement to change footprints within the factories left us with available space.
How we build chips, the different sizes of the chips and the different facility and
physical requirements of the equipment to build the chips actually pushed us
as a business into a substantially different factory footprint. Because of that we
eventually end-of-lifed this facility and basically moved and it sat empty for
many years.
Although the facility itself was used on and off for miscellaneous different internal
purposes for many years, Intel was having substantial compute requirement
growth needs globally. Our rate of growth in the Data Center from a space,
power, and cooling perspective was very, very substantial. As we looked at our
global footprint and where we needed to build based on where the right people
were, what the bandwidth and latency needs are for the needs of our facilities,
and what made sense financiallydo we green-build do we retrofit?we basically
did a survey of our global landscape and said Hey, New Mexico is actually a
good place for many reasons. Geographically, from a power perspective, from a
reliability and a cost perspective, it was one of our best locations globally. And
here we had this fabulous, improved facility that was sitting empty that we could
re-use as opposed to doing a green build at one of our other locations.
The lower cost and risk perspective was really great here, so we said Lets convert
an old factory instead of building a new facility. We have 150 sites globally and
we had a lot of opportunities to go pretty much anywhere we wanted around the
world but it made more sense financially to do a re-use of an old factory building
and specifically do it here in Rio Rancho because of the low cost of energy and
the high availability of power.
238
The fact that you were working with an existing facility, how did its
The Art of the Data Center
13: Intel
Figure 13-2 Air handlers on the third floor of Intels Rio Rancho Data Center.
We have multiple different types of compute within Intel. Some compute are
The Art of the Data Center
not highly critical but other servers if they go down our stock will go down or
our employees wont get paid; our finance and HR compute. High availability is
different than a lot of our high density compute internally. The largest percentage
of compute at Intel is high density that did not need that ultra-high reliability,
just reliable services.
So, as we were going through the design to build our higher density, we kept
asking ourselves what does that mean? Ultimately the cooling outlook was
determined to be, if we ran our Data Centers and completely separated the hot
air and the cold air and got away from hot spots or warm spots, we could gain
about a 24 percent mechanical efficiency at the cooling coil. Meaning if I had
1,000 servers, for example, in a room that was 5,000 sq. ft. (465 sq. m.) or if I
had 1,000 servers in a room that was 1,000 sq. ft. (93 sq. m.)the same exact
IT loadwe could run about 24 percent more efficient with a high dense type of
environment instead of spreading out the load. The tighter and the higher density
we could go, the more efficient we were.
At that time there was not really a product on the market to cool 30-something
kilowatts per rack without going to a liquid type of design, which was incredibly
expensive. We actually designed the chimney cabinet that exists on the market
today. Multiple vendors are selling it. We chose not to patent it so it would get
out to the industry, but we actually designed that at this site for this need. We
figured out that that was a great way to control the air completely and not let any
of the hot air mix with the cold air. You can do it through hot aisle containment
or chimney racks and there are multiple solutions today to accomplish this, but
at the time we basically invented the Data Center chimney rack and we designed
this room around it. When you walk in the Data Center today and you see many
thousands of servers in a very small space what you find is that there are no warm
or hot spots in the room because its completely controlled.
The mechanical design was to totally control the air, have no leakage of hot or
warm air into the room, and have the highest density to provide the hottest air to
the coil. Instead of mixing hot air with cold air (pre-cooling), we want it as hot as
it can get to the coil for the highest efficiency. There were some struggles. This was
definitely the highest density we had ever done. In fact, outside of a few super-
computers its still our highest density design five years later. But it was extremely
effective and worked very well for us.
241
13: Intel
Figure 13-4 Structured cabling and electrical infrastructure are routed overhead in Intels
Data Center.
Why did you forgo a raised floor in your high-density Data Center
design?
People do raised floors for a lot of reasons but the original intent of raised floor
was to distribute the right air to the right parts of the room to manage the hot
spots. We dont have hot spots because we 100 percent manage the air and we
let no hot air into the room. If you control all the air in the room then you can
dump the cold air anywhere you want and you can avoid the cost of building the
raised floor. As long as you have enough cold volume in the room to meet the
draw of the servers it doesnt matter where it gets in to the room. All network and
power is done overhead for simplicity and agility.
242
13: Intel
Figure 13-4 Structured cabling flows through an overhead cable tray in Intels Rio Rancho
Data Center.
When you look at the cost of building Data Centers per square
foot, from a design perspective, its going to be really hard for the
industry to justify building mega Data Centers anymore.
Coming with that higher density, is there a greater risk around thermal
runaway? If something breaks, things are going to heat up that much
faster. Or have we already crossed the point at which in most Data
Centers things are going to heat up very quickly anyway?
Thats a great question. Thats something that we experienced early on with our
really high densities. If you cool a Data Center in a very traditional manner
thermal runaway is definitely an issue that people have to consider. But theres a
multitude of ways of mitigating that risk.
I think what were going to find and what you already see starting to happen, if
you build high density Data Centers and you have alternative types of cooling
like free air cooling all of a sudden that doesnt become such an issue because the
245
normal mechanism of cooling your air is evolving. If you try to do high density
13: Intel
with traditional cooling, its a big gap to consider. I can tell you that as we went
through this process globally we learned our mistakes the hard way a few times.
When you have many thousands of servers in a very small square footage, you can
have a runaway condition very quickly if you completely lose all of your cooling,
for example.
How long did it take for the design and reconstruction of the building?
The design phase of the building was about seven months. A lot of that was going
back and pulling up 20-year-old plans and validating a lot of the old systems that
were in place and doing gap analysis of what we had versus what we needed to do
246
this high density, high performance installation. The entire project from design to
The Art of the Data Center
Figure 13-5 Intels Rio Rancho Data Center features 8 MW of electrical capacity.
Were there any surprises or challenges that arose along the way?
A whole bunch of them. Most of the gaps and surprises that we came up with I
would call normal construction project issues. They were mostly around taking
an old factory that was designed for extreme low density and converting that to
a very high density power delivery and cooling design. Having the right chillers
in the right places and the right air handling in the right places and how do we
deliver enough power to every rack to support the high density? There were just
some operational challenges. I wouldnt say anything major, but a lot of in-the-
field decision making on how to get over obstacles using an old building that
wasnt designed to do what were doing with it.
247
If you could go back and design the Data Center again would you do
13: Intel
anything differently?
Its all about the timing. So for example, from the power distribution perspective,
at the time nobody was doing densities like this and you couldnt get enough
power to the rack the way we wanted so we had to use vendors in some cases
and suppliers in some cases that we just werent completely happy with but
(did) to get the densities that we wanted. This was kind of an untapped market
to a certain degree so we worked with what we had or could get. So, I wouldnt
necessarily say that we would do things differently, but the timing was such that
we were leading the drive to go to high density and a lot of the products werent
in place.
The biggest challenge we had from a design perspective using this old facility
is going back to the point that traditionally the factories we build are designed
to have mechanical on top, factory in the middle, and electrical on the bottom.
What that means is we had water on the top of the compute and electrical on the
bottom. So making sure that we designed the pipes on the top to be contained
and have the appropriate amounts of leak detection was a must. We had so many
weld joints and so many solder points and so many pipe fittings, things were
going to leak at some point with thousands of fittings so we needed to make sure
leaks would not affect us.
With the super-high density and all of the technologies that come
with that, do you feel that these are universal solutions that can be
implemented everywhere or are some of these just corner cases that
worked here but if someone else were designing a Data Center maybe
it wouldnt be a good fit for them.
Thats a good question. For us this was our big jump into extreme high
density, we called it HPDC for high performance Data Centers. It has become
our universal standard for designing Data Centers globally now and with
virtualization, this will remain our standard for more traditional servers as well. I
think that certainly there is evolution in technologies and vendors improve things
and make them a little better so we see improvements year to year.
248
Any other lessons for people to consider from your high-density Data
13: Intel
Center design?
Historically, we had traditionally built Data Centers without the larger IT needs
clearly in mind. We have a team that builds facilities and theyre focused on
the physical stuffthe power, the cooling. The facilities have not always been
built with a direct correlation of the complete use model and it wasnt always an
optimal design.
What weve moved to and what you see in this design for the high performance
Data Center, we built a room to function as a large compute solution. When we
talk about Data Centers internally we dont talk about the facilities anymore,
we talk about the entire IT stack. We build our facilities in alignment with
the compute and the storage and the network pieces needed to support the
application layer. We are getting good about looking at the entire stack as a
solution with less focus on the individual pieces, including the facility.
reduce our cost and footprint in the Data Centers by investing the right money
The Art of the Data Center
in the right places. Thats why we really dont focus so much on facilities anymore.
Facilities are one component of the entire stack.
How does that manifest itself in your design? I think most companies
design the physical layer of their Data Center to be very reliable but as
essentially a self-contained set of infrastructure components. If youre
considering the entire stack, how does that drive down and impact the
physical design?
I think thats a really good question, and it took us a little bit to get good at that.
About 70 percent of our IT compute is designing next generation chips. Like
airplane and car manufacturers will design concepts electronically and model
them before theyre physically built we design our chip and model it before we
physically build it.
We basically take the business requirement/compute requirement and we associate
that to our high-density knowledge. (We know) we can get X amount of cores out
of a rack so we need so many racks to fulfill that design requirement. We build
around the requirement. We dont build the facility and see what were going to
put in it 10 years down the road. We figure out what we need and we build the
facility around it with the application layer clearly in mind. Three to five years out
is the target. Were not building 20-year facilities; the technology is changing too
quickly. So we try to be very adaptive. I think that goes back to the main point
of, we build modular for cost and agility, because technology is changing too fast
for us. We cannot build a Data Center today that is going to satisfy our need in
10 years so we would rather build small ones that meet the needs today and build
another small one or medium onewhatever you call 8,000-ish sq. ft. (743 sq.
m.)five years from now when the technology has changed and we know what
the 2015 chip and the 2020 chip is going to look like, for example. Our design
target should be based on overall uptime of the stack; over-building costs too
much money and under-building creates unwanted risk. When you consider the
entire stack concept, reliability can be delivered in many different ways.
251
And then you refresh the Data Center modules as you need to?
13: Intel
Bingo. Thats exactly right.
Any final lessons that you would like share about this Data Center
project?
Internally, one of the Aha! moments that we had through building this Tier II-
ish facility is a strong design can deliver great reliability at a lower cost. Weve had
no downtime in almost 5 yearszero. I suspect at some point well have some
downtime somewhere but what we are finding globally is you can build a Data
Center without all the components you think you need and just do it differently
with more intelligence and achieve the high availability numbers without
spending the money.
Were building two Tier II-like Data Centers and getting Tier IV performances
out of them. Really thats the sweet spot for us because were not spending
the money but yet were getting the reliability consistently with the lower tier
investment.
Figure 13-6 One of the air handlers supporting the Rio Rancho Data Center.
Essential Details
ORGANIZATION: IO
Location: Phoenix, Arizona
Online: Opened as a water bottling and distribution plant in 2006. Acquired by IO in December 2008 and
brought online as a co-location facility in June 2009.
Notable features: Time-shifted cooling. Thermal storage featuring polyethylene balls of ice and a glycol-
water solution. Patented cabinet enclosures. LED lighting. Ultrasonic humidification. Modular components.
Time to design and build: 6 months for the first phase.
Size: 538,000 sq. ft. (49,981.8 sq. m) of total space divided among two warehouses, including 360,000
sq. ft. (33,445.1 sq. m) of hosting space.
Power: 120 MW
Tier: III
Cabinet locations: About 3,000
Power Density: Varies
Infrastructure Delivery: Cooling is delivered through a 36 in. (.9 m) raised floor. Power and structured
cabling are provided overhead.
Structural loading: 700 lb. per sq. ft. (3,417 kg per sq. m)
Fire suppression system: High sensitivity smoke detection, pre-action dry pipe.
Those who live in extreme environments
Background
The Interview
IO began as a traditional co-location provider and has since expanded
into manufacturing modular Data Center components as well. How
does modularity fit into your Data Center strategy and what role do you
see it playing in Data Centers going forward?
It comes from a place where we believe there are four things that are
fundamentally wrong with Data Centers as theyve been constructed for a long
time. I come to this with a view that there has been what I would describe as
incremental innovation in the Data Center over the lastlets call it 35 years
or even longerfrom the late 60s when the first data processing rooms were
257
stood up for mainframes to today up until say a year and a half ago. They have
14: IO
essentially all been built the same way.
At the end of the day there are cooling elements. There are power distribution
elements. Its a room. Its part of a facility in one way, shape, or form. Yes, the
components have gotten more efficient individually. Theyve gotten bigger. They
have more capacity than theyve had before. There are more of them. There are
more air handlers than less. There are bigger chillers than less. There are variable
speed drives. But at the end of the day the overall architecture of the Data Center
hasnt changed in any meaningful way in a very long time. If you look at the
architecture of what we describe internally as Data Center 1.0, we believe there
are four fundamental flaws of that architecture.
First, its insecure, both physically and logically in most cases, meaning that the
control software and the control tools that have been implemented typically in
traditional Data Center builds have not been purpose built or designed for that
outcome. As you well know, most of these came from being building automation
systems or building management systems and not really from the industrial
controls world, which have been mission critical to it. They have come out of the
commercial building industry of managing HVA systems, managing office air
conditioning.
Then the physical design of the Data Center itself, because of the complexity
and the number of discreet components in itself makes it very difficult to secure.
You have lots of doors. You have lots of openings. In every Data Center theyre
different. In some cases, Data Centers have been placed in to buildings with other
uses, with people and other things. Holistically, we believe that the traditional
Data Center environment is not well suited for managing security.
The second major flaw of the traditional Data Center is that it takes far too long
to deliver. In the context of what they support, from an IT infrastructure both at
the physical layer up through the application layer or the virtualization layer and
then ultimately users, the speed at which IT moves is in months not years. So
they take far too long for most enterprises in our experience.
In the government and even for service providers that tend to be a little more
The Art of the Data Center
agile, from the day they first invent the idea of a new Data Center to the
day that they rack IT, its typically over two years to go through the whole
procurement process, decision making, analysis, engineering, all the way through
to commissioning and then finally a piece of IT being stacked. Because of that
length of time to construct, it also then forces that asset to try to be more relevant
for a longer period of time because you cant cycle it. It forces you to look at the
Data Center in a 10 or 15 or even 20 year event horizon for utility. Again, in the
context of the IT world every year you add youre adding an enormous amount
of variability and complexity in calculating exactly what its going to look like
one year, two years, three years. Then imagine trying to look 10 or 15 years out. I
always quip, Whats a 10 year old cell phone look like? It gives you a sense of the
technology challenge there.
The third component that is broken in the existing model is that Data
Centers cost too much. If you look at the total deployment from conception
to acquisition of whatever the particular real property component is, the
construction process all the way through to the non-IT fitup of the Data Center
to then the point where you start to add IT is very expensive on a unit basis.
Because its based in the construction worldconstruction I believe is the only
industry over the last 20 years that is less productive today than it was 20 years
agoyou have productivity regression. In addition, construction in and of itself is
subject to wage inflation and raw goods inflation and these sorts of attributes. You
see that theres not a visibility in to that unit cost going anywhere but continuing
to escalate over time.
Compare that to the IT world. Virtually every component of the IT stack is
actually going down on a unit cost and theres visibility to continue unit cost
reductions over time whether youre looking at storage devices or CPU cycles or
network capacity. All three over time decrease in cost. In fact, the productivity of
the IT department is the driving engine of productivity in the world today. IT
department budgets, if you poll IDC or Gartner, have been essentially flat for 10
years and theyre doing hundreds of times the amount of computational work
they used to do. It doesnt match up very well.
259
14: IO
Figure 14-2 Inside the first phase of IOs Phoenix Data Center.
The last component that weve attempted to solve with our technology platform
and we believe is a flaw in the 1.0 model is the lack of scalability. The other ones
are pretty easy to point at. Scalability is all of the things that we aspire to in the
IT stackmodular design, modular components, in-place upgradable, just-in-
time deployment. All of these attributes that we see in an EMC storage array or
we see in a Cisco router platform or that we see in any of these other devices that
make up the IT stack and the virtues of those devices that we as IT consumers
have grown accustomed to having access to dont show up in the Data Center.
I boil it down to one basic concept from scalability. Everything in IT now is
thinly provisioned. You have the ability to make very quick decisions and to
provision an asset and then re-provision that asset, change its capacity rapidly
and redeploy it to the same user, whereas the Data Center is a fixed provisioned
asset, typically. You make a varied set of discreet engineering decisions up front.
You apply discreet engineering principles to those and you deploy an asset that
hasIll boil it down to one metric that is the quintessential thick provisioned:
this Data Center is 150 watts per square foot (1,614.6 watts per sq. m). Youve
now made a decision that impacts everything in the engineering.
260
This is the part that most people dont get and why I think theres been so many
The Art of the Data Center
challenges in the Data CenterIT people dont think that way. IT people
provision a router and then assume that if they need more capacity they can pull
a blade out and put a new blade in that gives them more capacity. So when they
buy a Data Center or use a Data Center that someone says, its 100 watts per sq.
ft. (1,076.4 watts per sq. m) the assumption is that Oh, two years later when
I need 150 watts a foot (1,614.6 watts per sq. m) I should be able to just slide
a blade out and put another blade in and have that capacity, whereas we know
in an engineered, construction-based Data Center you just cant do that. Its an
integrated system. When you design x amount of power distribution that means x
amount of cooling. When you say x amount of redundancy you get x amount of
redundancy. It isnt that simple (to change). Hence why it takes so long.
The other component of scalability that I point to that weve attempted to solve is
the ability to variably match the service level goals of the userthe applications
inside the Data Centerwith the resources being deployed. The other very
challenging decision thats made in Data Centers today that affects scalability
is you have to essentially decide what the resiliency tolerance you have for the
entire Data Center. As we know, its very difficult to have part of a Data Center
in the Data Center 1.0 world to be N+1 and one part to be 2N and one part
to be (something else). Or, more importantly, to have one part be five nines (of
availability), one be four nines, one be three nines, and one be who-cares.
So, if you take those four things, thats the state that we believe the current Data
Centers are in. Having been in the Data Center world for a decade and having
been in IT and technology for a decade and a half, we came to the conclusion
about two years ago when we finished constructing the first phase of this Phoenix
Data Center that this just wasnt going to be sustainable over the next 20 years.
That continuing to build Data Centers, even though we build them much larger
and we scaled them up and all these things. These four problems were going to
adversely affect, if not become untenable, going forward for the IT consumer as
we continue to see IT demand grow. The demand for IT cycles is exponentially
increasing with no end in sight and in fact accelerating, I would say. Which
means more and more and more Data Center, which if we keep building the way
weve done were going to paint ourselves in to the proverbial corner.
261
With that as a backdrop, thats the way we see the world. What we attempted to
14: IO
do then, and what we believe weve done in our modular product, is to solve those
problems with two fundamental principles. Making the Data Center delivery
component itself a manufactured and engineered component that is modular
in aggregatefinitely modular and as a system modular. You can scale up
vertically and horizontally. So you can go inside the module and you can upgrade
components; you can in-place upgrade power delivery and cooling infrastructure
inside the thermal dynamic module. You can also then add more modules when
you need more capacity in that vector. And manufacture that so you get all of the
benefits of having a supply chain, having innovation cycle over and over and over
again rather than having all of the engineering skill that goes into a Data Center
essentially go in to this Data Center and then moves on with the construction
or engineering firm to the next oneand theres no incentive really to drive that
innovation cycle over and over.
The real incentive for most engineering firms is to take the same plans they
already spent the money on to build and just hand them to the next guy and
charge him the same amount. Whereas in a manufacturing process were incented
and our customers are incenting us to manufacture better, faster, cheaper just
like everything else. So cars today for essentially the same price as 15 or 20 years
ago have 3x the functionality. From air bags to ABS, to all these things, are
innovations that have shown up, and quality and cycles are better.
The second part we have attempted to solve in the approach to modularity in
solving these four problems, is that now that you have a standardized delivery
unit thats manufactured, on which to layer on top of that a software automation
layer or control layer. In our case we call it a Data Center infrastructure operating
system. That then gives you the ability to manage and actively and pro-actively
control and then ultimately intelligently optimize that physical layer to match up
with the IT stack.
262
So, taking those four problems into consideration we determined the way we were
The Art of the Data Center
going to solve those problems was to manufacture the Data Center components
and then software automate them. In getting there we now have a 300,000 sq.
ft. (27,870.9 sq. m) factory that we operate where we produce these modules.
Each module is roughly 200 kW of capacity. We have over a hundred frames in
production. Weve delivered over 70 modules to customers at our own locations
in Phoenix and New Jersey where we host the modules for customers.
The punch line is, weve already recognized for customers anywhere from 50 to
60 percent reduction in initial capex in apples-apples environments. And weve
also seen ongoing operating expense reductions in the order of 25 to 30 percent.
In addition to that, were able to deliver quickly. Just as one discreet example,
Allianz, the German insurance company, we delivered a 3.6 MW Data Center
system to them in 91 days from the day they signed the contract to the day they
racked IT. So we symptomatically now are seeing the results of this platform
solving those four problems.
Lets talk about the Phoenix site in particular. It seems this started as
a conventional space and it has gone to using modularity throughout
the facility. When did it first open?
We acquired the site in December of 2008. Its an interesting site in the sense
that we repurposed it. It was previously a distribution and water bottling facility,
owned by a company that had subsequently gone out of business. The structure
and the site was developed and turnkey delivered to the water bottling company
in 2006. We acquired the site in December 2008 and built out phase one, which
is lets call it the first half of the site. Its a 538,000 sq. ft. (49,981.8 sq. m) facility
and its on about 40 acres (16.2 hectares ) from a campus. It has a substation on
site provided by the local utility.
We built out the first phase in what we call Data Center 1.5, which is a scaled
deployment of capacity, bringing to bear all of the things that we thought
was necessary to start to solve these four problems in a scaled way in a more
traditional technology architecture. Its about a 250,000 sq. ft. (23,225.8 sq. m)
raised floor space. Its all in one room, which is kind of unique, you dont see very
often. Very high ceilings, 30+ ft. (9.1 m) ceiling heights in the space. We have
24.5 MW of UPS (uninterruptible power supply) online net of redundancy in
that phase one space. We moved our first customer into the site in June of 2009.
263
We have 8,000 tons of chiller capacity, two 4,000 ton plants. This is phase one
14: IO
Im speaking only to. We have two 16 MW Caterpillar generators through two
sets of 20 MW paralleling equipment that power the phase one space exclusively.
Its broken into four individual pods, but again all still in the same room. Each
pod yields about 45,000 sq. ft.(4,180.6 sq. m) of usable raised floor. So out of
that 250,000 sq. ft. (23,225.8 sq. m) in the phase one Data Center room theres
180,000 sq. ft. (16,722.5 sq. m) of net usable raised floor for IT.
We deployed a service gallery architecture so down the center of the room and
to the north and the south sides of the room there are service galleys that have
the power distribution power distribution units as well as the CRAH (computer
room air handler) units located in those. Its a full chilled water system, bi-
directional loops. We have a Tier III design certification for this site for phase
one from Uptime Institute. One hundred percent concurrently maintainable
across the entire system architecture, both power distribution as well as cooling
infrastructure. We have a meet-me room on site, actually two that we operate.
Today there are about 20 different telecommunications providers that are here
on type 1 diverse fiber builds into the site. So its a very large scale, kind of
traditional-plus Data Center build that we completed.
Figure 14-3 Phase one of IOs Phoenix Data Center is a traditional build with power and cabling
routed overhead and cooling delivered below a raised floor.
264
How long did it take to convert the building from its original use as a
The Art of the Data Center
I confess I hoped that your Data Center retained some of the water
bottling infrastructure, so anyone working in the room who wanted a
drink could easily grab one off of a conveyer belt somewhere.
Everybodys like Water bottling in Arizona? Whats funny about it is, Arizona
actually has really cheap water and has really cheap power. Because of that, the
site was obviously also very well suited for a Data Center. We use a tremendous
amount of evaporative cooling in our hydronic economization strategy so we
consume a lot of water.
In addition to that, low cost power and high resiliency power is valuable as well.
Arizona turns out to be a very, very good location for Data Centers.
I was going to ask what caused you to place your Data Center in
Phoenix.
We have looked at obviously a lot of different places. We like Phoenix for its
proximity to California from a commercial perspective, because there are lots
of Data Center consumers and businesses in California. And Arizona, Phoenix
specifically, is a short trip from Los Angeles.
In addition to that, from an environmental perspective or natural disaster all of
the things that California is exposed to Arizona isnt. So earthquakes primarily,
coastal risks, those sorts of things. Its a very, very stable environment in Arizona.
Its also very dry most of the year, which makes the ability to use pre-cool and
direct cooled hydronic economization very, very effective in this market.
265
The cost of power is very stable here as well. The power distribution in Arizona
14: IO
is still regulated by the state. Though it may not be as cheap as it would be in a
competitive market, its still very, very low cost because of the amount of hydro
and nuclear power that feeds Arizona. In addition to that, because of regulation
it provides great stability in pricing because the process to change rates requires
government action. It yields pretty stable rates.
The only negative obviously people point to and ask us about, is, well, its hot
very hotduring many parts of the year. But envelope load in a Data Center
of this size, meaning the cooling load derived from the exterior heat, is not a 3
percent issue in a Data Center. And the bigger the Data Center the less of an
issue it is. The dry environment more than compensates the additional cost that
you would have to your cooling load during the summer months. And as we
know, for a Data Center with 24 MW of IT load in it, it doesnt matter what the
temperature is outside is its going to overheat if the cooling turns off.
the legacy-think. Oh, well the standard says this. Weve all see IT stuffed in a
The Art of the Data Center
cabinetnot an IT cabinet but under a desk. Itll be running 120 degree (48.9
Celsius) air and its perfectly happy.
With the amount of abuse that people subject laptops to, it does seem
that Data Center hardware could withstand more severe conditions
than they are typically subjected to.
Especially since its the exact same gear for the most part. At least in your PC
server environment, the Wintel boxes, there is not a significant amount of delta
between commodity Wintel servers and whats in a desktop PC or a laptop.
Its essentially the same, maybe a faster processor. Same manufacturer. Same
architecture.
Where this comes from, though, is one of the things that weve attempted to
address in our technology platform. Theres no integration between the IT
equipment and the Data Center in the legacy build. And I mean that from a
programmatic or software management perspective. There are very few places
where you can actually correlate the performance of the IT hardware with the
performance of the Data Center. Whats the temperature of the chip set? Whats
the temperature of the room?
What weve attempted to do and have done now in our IO. Anywhere product
with our IO.OS on top of it is, you can go in and in one of our modules you
can set an average planar temperature for IT to be the set point for the cooling.
Rather than cooling the room you cool the chips, because the chips are what
youre trying to maintain.
I read about plans for this Data Center to have a massive solar panel
array300,000 sq. ft. (27,870.9 sq. m) of panels. Has that been
installed?
No, it wasnt, actually. None if it has been installed because the state of Arizona
abandoned their incentive structure for solar and without the solar incentives it
did not pencil.
267
Covering an 11 acre (4.5 hectares) roof with solar panels yielded less than 3 MW
14: IO
of peak power production during perfect operating conditions. This site pulls
north of 15 MW of active demand, 24 hours a day, 7 days a week, 365 days a
year. And from a kilowatt hour production perspective it was going to be less than
an 8 percent contributor to our total power load here on site, if we kept all the
power on site. When you look at the ROIC for the cost to deploy, if you try to do
it without incentive programs its a long ways from penciling. Especially in a state
where I can purchase power for 5 cents a kilowatt hour from the utilities. It gets
very, very, very challenging for solar to keep up with a scale, nuclear power and
hydro.
On paper, this site seems to be a perfect test case for using solar in
a big way. You have the option to employ it on a significant scale and
youre in Phoenix where sun coverage is ideal. So if its impractical for
you, is solar just not a workable solution for powering Data Centers?
I think energy density is the primary problem. The amount of solar panels
you would need to provide a significant enough load to actually create a valid
alternative just isnt practical. And thats before you get to the cost analysis. If 11
acres (4.5 hectares) yields 3 MW, for what we would need here would be 5x. We
would need almost 50 acres (20.2 hectares) of solar panels to match our peak
demand. It doesnt really work physically from an energy density perspective.
The second thing is, when you get to the cost side of it, the cost per kilowatt
hour fully amortizing the cost of the panels, the installation, the maintenance
and everything else. And then appropriately assessing the panel risk, because if
you go with the newer technology from a cell and panels it hasnt been in the
field long enough to know what the degradation rates going to be over time on
it. So your effective yieldtoday you may be yielding 100 GWh per year and
if the degradation rate isnt 3 percent and its 12 or 15 percent, 10 or 15 years
from now that production has gone down 15 or 20 percent, which can massively
impact the return on investment calculus. So with that you have two choices.
You either abandon the newer cell technology, which is higher density and lower
cost, and go with a more traditional cell technology, which has been proven and
there are definitive, quantitative analysis and actuarial analysis around what the
degradation is over time. In that case youre now paying too much and youre
using older technology. The more we dug into itand I spent a lot of time
personally working on thisits a very challenging analysis.
268
I think your assessment is dead on, that if it didnt work for us in this
The Art of the Data Center
environment (it wont work for any site). And were not shipping any of the power
off-site. We were going to invert all of it and put it right here on site and use it all
ourselves. Without a relatively massive tax subsidy it doesnt work.
I have seen some solar arrays deployed at Data Centers but theyre
just a fraction of the facilitys overall power capacity.
Yeah. Its neat to point at, and its a neat technology, and I guess its one to at least
start looking atrenewable on site and these things. But I think for industrial
use, which is what Data Centers truly are is an industrial use of energy, its going
to be pretty challenging for it to meet the density needs. Those silicon chips use
a lot of power per square millimeter and it doesnt make a lot of sense if you can
match it up with things that produce a little bit of power across square meters of
capacity.
14: IO
Figure 14-4 IO creates ice during the night, when energy rates are lower, and then uses that ice to
cool the Data Center during the day. This use of thermal storage saves the company
millions of dollars per year.
The other barrier to it is being willing to operationalize it and really put the time
The Art of the Data Center
and resources into building the automation necessary to take full advantage of
itand then also having engineering and analytical skills inside your organization
where you can keep on top of it. But it well in order pays for itself. Weve saved
millions of dollars over the last two years.
Some new Data Center designs arent using raised floors. What
14: IO
prompted you to use a raised floor and this cooling distribution
method, compared to other options?
We had a little bit different decision making criteria than others do. Because
we dont know exactly what the customer fitup is going to look like because its
a co-location site. The raised floor provides a tremendous amount of flexibility.
Anywhere you can take a tile out you can put one in to push airflow and so it
gives you tremendous flexibility in where and how the individual cabinet rows
and IT kit end up being laid out. Ive built and operationalized Data Centers
with no raised floor and it works fine. I dont believe theres any real operational
benefit one way or the other, other than the flexibility you get in air movement.
Where you have a room where youre blowing air essentially without raised floor
and youre distributing it you have to lay out the IT kit ahead of time. Whereas in
an environment where you have a raised floor plenum like you have here you can
really make ad hoc decisions about where youre going to place IT kit.
Theres a tremendous amount of value in having flexibility. Any time you can
eliminate construction inside a Data Center youre eliminating an enormous
amount of risk.
Weve talked a lot about the first phase of the facility. Lets discuss
the design of the second phase and its use of modularity.
The second phase of the site is now essentially the second warehouse. The first
250,000 sq. ft. (23,225.8 sq. m) warehouse we turned into phase one. The second
250,000 sq. ft. (23,225.8 sq. m) warehouse is now a bidirectional chilled water
spine that runs along the west wall of the warehouse. That spine then feeds
272
modules that we set down. Also along that spine is a set of medium voltage unit
The Art of the Data Center
substations that are tied to a 20 MW generator farm. The chiller capacity is tied
to a 4,000 ton chiller plant in phase two that does both hydronic economization
as well as traditional water evaporated, water cooled chiller. The power spine
and the chilled water spine then connect to power modules or network or data
modules. The power modules are 2 MW, integrated, manufactured, energy
storage and distributionso, UPS, battery and distribution panels that are in a
module. You set down two of those modules which gives you the ability to up to
2N resilience for the customer. You set down a network module which is your
meet-me room or carrier interconnection space for the system. Then we add data
modules to the end of those three modules until we get to 20 and then we start
the next row. Its a very, very different approach than you would see otherwise.
Its actually really cool here. Whats great in our tour, when we take customers
through, you can actually walk from what was legacy, traditional Data
Center design and deployment and then somewhere in between that with the
ThermoCabinet architecture and things like that, and then you can walk into
what is DC6 and DC7 which is the modular. And its a completely different
experience.
The fire suppression in that room is just traditional wet pipe EFSR (Early
Suppression, Fast Response fire sprinklers) because you dont need to put dry pipe
in because the modules are each individually watertight as well as theyre each
individually fire suppressed. And its a steel box. Its fire-rated as well. Each box
has its own fire detection and suppression system which is gas based, inside each
of the modules.
Each module has all the air-handling capacities, so one thing thats immediately
obvious when you walk into the modular is that its quiet because all the air-
handling capacities are inside of each module. When you walk into the larger
room its dead silent. And then you walk into the module, and then the air
handling capacity is modular, in and of itself and is inside of each module. The
power distribution PDUs are inside of each module. And then you have up to
20 racks of IT equipment per module as well. And those modules run from 200
kW to north of 500 kW depending upon what the customer requirements. The
minimum threshold is about 10 kW per rack and then can scale to well north of
25 and 30 kW a rack, depending what the customers needs are.
273
14: IO
Figure 14-5 IOs use of modular components reduced costs, quickened deployment times, and
improved scalability.
Were there any surprises along the way, during the deployment of
either of the Data Centers different phases?
Unfortunately, one of the attributes of having done this for a long time is that
the surprises are no longer surprises. Theyre expected. How they happen I think
sometimes surprises you.
Ive been surprised as weve deployed the modularany time you deploy new
technology and a new way of doing things, especially in as risk adverse of a
marketplace as IT professionals tend to be, though theyre tasked with technology
which is always innovative and always new they tend to be very careful about it
how quickly customers as we show them the traditional way of doing it and the
modular, how quickly theyre willing to move to the new way.
274
Ive also been surprised at the operational efficiencies that weve picked up that
The Art of the Data Center
I didnt fully appreciate with the modular approach versus traditional. Most
important in that is the standardization. All of the components are the same.
Theyre in the same spot. Theyre manufactured the same way. That you can
blindfold a technician and take him from one Data Center module to the next
one and its exactly the same. What has surprised me is not that its standardized,
because we knew that when we built them. What it has resulted in is the
operational utility of a Data Center operations tech has been greatly, greatly
increased.
I was also surprised at how quickly this very large Data Center was filled. I
continue to every day benot shocked because its not surprisingimpressed at
how quickly were building IT infrastructure. It used to be business drove these
things and I think now what were seeing is consumers are now taking over the
drivers seat in technology. Its showing up in businesses but it is being driven
by consumers. Technology like the iPad and the iPhone and the consumer user
experience are driving business and commerce to keep up. And what its resulting
in is just an enormous amount of capacity being deployed and very, very rapidly.
Phase one here, 180,000 sq. ft. (16,722.5 sq. m) of raised floor and 24 MW of
UPS we essentially took to 100 percent utilization in 16 months. I think that
probably during the whole 60s and 70s and 80s there wasnt much more than
that deployed. Its an enormous amount of capacity. Then when you think about
the computational capacity of 24 MW, its pretty extraordinary. And this is one
Data Center.
Technology like the iPad and the iPhone and the consumer user
experience are driving business and commerce to keep up.Its
resulting in an enormous amount of (Data Center) capacity being
deployed and very, very rapidly.
275
14: IO
and other technologies to make their Data Centers more efficient but
their demand for capacity continues to grow. I jokingly call it the diet
soda syndrome: its half the calories so people drink two. Data Center
efficiencies free up capacity, but companies immediately consume
the savings. Theyre doing hundreds or even thousands of times the
computing than they did before but arent reducing their consumption
of capacity.
Theres a famous maxim that came out of the Industrial Revolution in the late
1800s. It basically says that humans are pre-ordained to take productivity and
turn it into more work. Efficiency never shows up in a reduction in use. It shows
up in more use. To exactly your point.
I can never remember the gentlemans name. I think he was a professor. His whole
assessment was, now weve mechanized everything and we just do it all faster. We
just do more. No one used the telephone to reduce the amount of time that you
talk to people. You talk to more people, right? I always point that out to people.
Its the same in the Data Center.
Back in my early days when I was in IT at a very fundamental level, the old joke
was the development server. There is no such thing because the minute you plug
it into the wall its in production. The minute you give a developer access to a
device that users can connect to its going to be in production before you know it.
I think similarly weve done the same thing in Data Centers.
The design elements of this Data Center have obviously evolved over
time. If you could go back in time to when you started with the site is
there anything you would do differently?
We no longer will even do traditional Data Center build anymore. Its all
modular. And after seeing it and deploying it and operationalizing it and all these
other things at this point in time and knowing the underlying economics of it
and whats its done for us from a commercial perspective, I would never go back
to building a traditional Data Center again.
276
Is there any final advice that you would offer someone if theyre
The Art of the Data Center
Figure 14-6 Overhead structured cabling bundles in IOs Phoenix Data Center.
277
14: IO
Figure 14-7 Standby generators provide backup power for the Tier III Data Center.
Figure 14-10 Fire suppression controls and a badge reader adorn the side of a Data Center module.
14: IO
279
Figure 15-1 NetApps Data Center in Research Triangle Park, North Carolina was the first to earn
the Energy Star for Data Centers rating from the U.S. Environmental Protection Agency.
Images provided courtesy of NetApp.
Chapter 15
NetApp
Essential Details
ORGANIZATION: NetApp
Location: Research Triangle Park, North Carolina
Online: March 2009
Notable features: Pressure control governance of airflow to cabinets, airside economizer, data hall cold
rooms, 52U racks, rotary UPS system. Energy Star-rated Data Center.
Time to design and build: 19 months
Size: 125,000 sq. ft. (11,612.9 sq. m) total building space, with 33,000 sq. ft. (3,065.8 sq. m) of
hosting space
Power: 25 MW
Tier: III
Cabinet locations: 2136
Power Density: 12 kW average per cabinet, 42 kW maximum
Infrastructure Delivery: Power, cooling, and cabling infrastructure are delivered overhead
Structural loading: 185 lb. per sq. ft. (903.2 kg per sq. m)
Fire suppression system: Double-interlock dry-pipe
Pressure, as the saying goes, makes diamonds. Air
Background
The U.S. Environmental Protection Agency established the Energy Star rating for Data Centers in 2010. Facilities
Note
are scored on a 100-point scale and earn the Energy Star designation if they are among the top 25 percent of
their peers in energy efficiency.
The Energy Star Program, begun in 1992 to highlight energy efficient products, was initially applied to
computers and monitors and later expandedinvolving participation from the U.S. Department of Energy
and applying to office equipment, major appliances, lighting, home electronics, new homes, commercial and
industrial buildings, and more.
283
The Interview
15: NetApp
What role does this Data Center serve for NetApp?
When we first set out to build this Data Centerits a greenfield site on an
existing parcel of land where we have some office buildings in RTP (Research
Triangle Park)the purpose was for it to support our internal research and
development, so (it was to be) more of a lab environment. A lab environment is
a little less rigorous in terms of infrastructure redundancy than a corporate Data
Center.
But shortly before occupying the building, shortly before its completion, which
was around January 2009, we were asked to retrofit a portion of it for corporate
IT. So one of the things I think that is interesting or unique about the Data
Center is that its a multi-tenant Data Center. It has corporate IT, which is high
availability with fully redundant on-site power systems with UPS (uninterruptible
power supply), and thats physically about one-sixth of the Data Center.
Comparatively speaking its a relatively big Data Center. Its not the biggest in the
world but its about 33,000 sq. ft. (3,065.8 sq. m) of cabinet space and capacity
of about 25 MW of equipment load. Thats pretty good size and the corporate IT
takes up one-sixth of that. So about 360 of the racks in the buildingthere are
2,200 racks, roughly, in totalare for corporate IT and the balance of the racks
or about 1,800 are for product development R&D effort to support our product.
Like you say, the R&D environment is a little bit more dynamic than the
The Art of the Data Center
corporate Data Center and the change control requirements are obviously a little
tighter for the corporate Data Center but from a Facilities perspective we tend to
do a lot of the same things. Were still concerned about lets say transformer load
balancing loads compared to the rated ampacity of the breakers.
We dont treat them terribly different although our experience is that the
Data Center people are more disciplined in terms of if you tell them how the
infrastructure works theyre going to adhere to it versus the R&D guys, for them
to string cabling in a weird way or to connect things that are fed from different
power services in a way thats not the greatest is more common just because there
are a lot more people youre dealing with. Thats how we deal with it. It hasnt
been too much of an issue with having two users.
15: NetApp
Figure 15-2 Cold room enclosures isolate hot and cold airflow for greater efficiency in NetApps
Research Triangle Park Data Center.
Because you were able to easily qualify did the fact that you wanted
it to be an Energy Star-rated site have a significant impact on the
design?
No, because the design really pre-dated any information we had that they were
going to roll out a program on the Data Center. Back in 2006 we actually
developed a fairly high density Data Center in one of our existing office
buildings. It was a 3 MW facility and the average watts per square foot was
around 500 (5,382 watts per square meter), which was fairly high, 8 kW per rack.
We were able to achieve that by enclosing the cold aisle and using at the time a
fairly unique differential pressure control scheme for managing the air that was
delivered to the cold room. This continues to be a pretty unique approach in
the industry in terms of feeding only the quantity of air to the cabinets that the
286
racked equipment is going to actually consume. That enables us, together with
The Art of the Data Center
the direct outside air/free cooling, to keep our PUEs really at a minimum.
Back when we built a prototype lab we had proved the concept of metering
exactly the amount of air to the cabinet as required based on differential static
pressure and then when we built this new Data Center what we wanted to do is
increase the density to 770 watts a foot (8,288.2 watts per square meter), which
was 12 kW per cabinet. We actually can do up to 42 kW in a cabinet with the
pressure control approach. The other thing we wanted to do was get the air
handlers closer to the load, so rather than them sit at the ends of the hot aisles
they were moved to directly on top of the cold aisles or cold room, with no duct
work. Then we added the outside air/free cooling capability which enables us to
cool the equipment about 70 percent of the year with no chiller.
Really, the desire was to make it more energy efficient and a higher density. The
thing we found out by building a vertically-oriented building and compacting
25 MW in a 1 acre (.4 hectare) footprint our construction cost wound up being
about a third what the Uptime Institute says it should cost for a Tier III Data
Center. From a total cost of ownership this facility, both because it has a low PUE
and because our construction cost was about $7,000 per kilowatt for Tier III, the
cost as we measure it internallywe look at cost per kW per month delivered to
our userworks out to about $60 per month for corporate IT here. That is about
one-tenth the cost of what we pay at a co-lo and its significantly less than what
we pay internally in other sites where we have Data Centers that we have built
and own within NetApp.
and assuming that its only going to go through a 10 degree rise on the coil. What
15: NetApp
happens is you oversupply, like you say, the equipment and you just get a lot of
mixing and turbulence. We deliver everything from overhead. The more air you
try to jam to the cabinet the more entrainment of hot air you get from the hot
aisle when you dont have any physical separation between the cold and the hot as
weve done with the cold rooms. What we basically do is put doors and a little bit
of fascia above the cabinets and drop the ceiling down. The construction cost is
miniscule compared to the mechanical and electrical.
Thats how we got there. We said, This is ridiculous. Even Liebert units and some
other custom type air handlers, the coils are designed for 16 degrees or so and if
the equipments seeing 20 youre missing it by a pretty wide margin.
Im not out there every day looking at Data Centers that have containment but
The Art of the Data Center
everything I read online tells me that they probably do have an issue with pressure
and flow. If theyre doing it with an air curtain, lets say, probably their curtains
are flapping open or whatnot.
So, to answer your question, I hear a lot as people come through the facility and
we have hundreds that come through here in a years time, organizations that are
customers or partners or prospects. They talk a lot about temperature control and
they look at what weve done and its an aha to them that this is the way it should
be done or one way that is an effective way to do it.
Figure 15-3
Pressure
differential
transmitter used
in NetApps RTP
Data Center
for cold room
pressurization
(right) and
mezzanine
pressure controlled
relief air (left).
with a 20 degree riselets say its 85 (29.4 Celsius) outsideit still makes sense
15: NetApp
to take 85 degree (29.4 Celsius) air and cool it to 75 (23.9 Celsius) because the
air coming back is 95 (35 Celsius), right? Obviously we want to make it as warm
as we can but the limiting factor, like I said earlier, becomes how warm can you
make it in the hot aisle before people get too hot?
We found that 95 degrees (35 Celsius) is about the limit, and particularly when
youre bringing in outside air in the summertime that has a fair amount of
humidity in it. It tends to get pretty warm. In fact, even though its cooler in the
winter and we could make a 60 degree (15.6 Celsius) cold room if we wanted, our
folks working in there would get used to that 80 degree (26.7 Celsius) air in the
hot aisle and then they would really complain a lot. And 95 (35 Celsius) in the
summertime is certainly not an OSHA issue and if you dress appropriately its not
a hazardous condition. Its more a personal preference. We try not to float things
too far down in the winterwe might go down to 70 (21.1 Celsius) because its
still free and then well float up to 75 (23.9 Celsius) in the summer.
The other thing that were doing is by having higher temperatures we never really
try to get to the dew point where were taking moisture out of the air, which
from a Data Center perspective the equipment doesnt really understand latent
cooling and humidity as long as you dont have it too high or have a condensing
environment. Our objective is not to do any condensing at the cooling coil and
waste more energy.
Again, if you look at the typical Data Center thats running a 55 degree
Fahrenheit (12.8 Celsius) supply, about a third of the work being done could be
latent cooling, or at least the rated capacity of that cooling coil is based on about a
third of the work being latent cooling. Thats probably not the case because most
Data Centers are built pretty tight and theres not a lot of moisture infiltration
but theres going to be some because youre introducing outside air for ventilation
and whatnot. Those are really the drivers for why we picked 75 (23.9 Celsius).
The other thing is, if you get up to 80 (26.7 Celsius) what you see is a lot of the
equipment the way its built will actually internally speed up the fans and the
fans can be a fairly high proportion of the energy used by the racked equipment.
We havent had a chance to study it to a high degree but we know that the
incremental energy used, once we get to 80 (26.7 Celsius) and we start triggering
these fan speeds up, offsets the gain we get on not running the chiller.
290
Is the design of this Data Center a standard for NetApp going forward?
The Art of the Data Center
Are the technologies that are being used repeatable at other locations
or are they specific to this particular site and its climate and other
conditions?
Thats a really good question. In 2006 we built this prototype room with the
differential pressure control and high density and then shortly after that we built
a lab in Sunnyvale, California, based on that concept and added a free cooling
capability to it. It was sort of the next generation to what we did here in 06.
They have some issues there with trying to get the exhaust air out of the building
because it was built within an existing office building and they had to create
a chaseas you bring in outside air you got to get rid of air youre not using
anymore from the hot aisle.
There are a few hiccups but this building, because it was a green field, we didnt
have that issue. The whole second story of the building is the air handling that is
able to get the air out of the sides of the buildings as we bring raw air in from the
roof.
So, to address your question, would we do (this design) in any location we choose
to build another Data Center? It depends, because some locationsit gets back
to the issue of using ambient air for free coolingif you have salts, if you have
corrosives, if you have high particulates, maybe not such a great idea. This is a
great location for doing free cooling in that were in a suburban business park
with pretty good air quality. The only issue we run into about a week a year is
pollen. Not that pollen is difficult to filter, but pollen is just pervasive and we
found that it gets around some of the filter rack. One thing we did here is we used
a commercial type air handler and saved a lot of money by using it and it does
by and large a pretty good job but the filter rack that the filters sit in have a lot of
bypass so weve had some issues with the filtration of the pollen. That being said,
its probably cheaper to run the chillers one week a year than it is to change 3,500
filters.
If you could go back and design the Data Center all over again what, if
anything, would you do differently?
Functionally, the only thing we would do differently is to add a pre-filter to the
filter rack to address the pollen issue.
291
Theres probably a half a dozen things we could do to make the building work a
15: NetApp
little bit better. Theyre so minor in some respects that I dont think it would be
meaningful to really go through them.
On the other side of it theres a few things we could do cost-wise to even reduce
the cost beyond where we have it today and still have the functionality. But the
building, when we built it, the total cost was around $66 million and we had less
than 1 percent change orders. So, it was a pretty interesting process and I would
say by and large we havent really done anything in two years to address any issues.
Were wrestling with this filter-rack thing now and its probably a couple-hundred
thousand dollar type of fix but when we started the whole process actually back in
March of 2007 (when) I was told to build this building I interviewed some name-
brand Data Center consultants and each of them that came here wanted to build
a single-story building about 15 acres (6.1 hectares) in size to handle 25 MW.
At the end of the day we built this 1 acre (.4 hectare), three-level building and we
did it with a local architect and a design firm that does a lot of biotech, because
to me it was more of a high-volumes-of-air, big-chilled-water-plant design issue
for mechanical/electrical plumbing than it was a Data Center. We actually used
a general contractor that had never built a Data Center, somebody that we had
worked with on some office fit-ups. The message there is we didnt buy into any
pre-conceived notions. NetApp was very involved as the project manager of the
building and every decision that was made.
We didnt even know for sure what it was going to cost us. Were not in this
business, right? We just used the prototype building costs and sort of said This
is going to be four times bigger and were going to need a shell and we think its
going to cost x. Prior to the modification for the corporate Data Center we had
a $45 million budget not including the racks and PDUs and we brought it in at
$44 million. It was just an interesting project, but not knowing what it would
cost every step of the way we looked for opportunities to save in the construction.
At the end of the day, functionally it has worked (with) very high availability and
low cost of ownership.
Data Center might be trying to get to 7 or 8. We were doing 12. By using the
The Art of the Data Center
pressure control we were able to get the higher density, the 770 watts a foot
(8,288.2 watts per square meter). Being able to get two or three times the kw per
rack that most Data Centers can get shrank the footprint and then by going with
a three-level building where the air handlers sit right on top of the load and the
utilities are all in the basement, you divide that by two-thirds. Those two things
really enabled us to get from 15, 17 acres (6.1, 6.9 hectares) down to one
(.4 hectare).
Figure 15-4
NetApp installed
commercial grade
air handlers rather
than customer
Data Center
models, saving
about $10 million
in capital costs.
I saw that you scored 99 out of a possible 100 points on the Energy
Star scoring. First, let me say congratulations. Second, though, I have
to ask: what was the one point that you missed?
Its not a point based system, its a performa that they create. Weve gone back in
and put in our numbers today and were at 100 based on our PUE being lower.
293
15: NetApp
Center metricPUE, LEED, et ceteraby doing things that dont
truly make their Data Center more efficient, but was there any point in
your project where you had to make a decision whether to incorporate
certain design elements to achieve a higher score versus its cost or
merits?
Thats a good question. Really the thing that drives the cost effectiveness of the
Data Center is the airside economizer. Its going to give you a lot more free
cooling hours than a waterside economizer. There are pluses and minuses to each.
If your ambient air is not good quality or you could use one in combination with
the other, but we know at 25 MW even in a moderately priced utility state like
North Carolina the free cooling, the PUE difference between lets say a 1.8 and a
1.2 is $6 million a year and thats going to buy a lot of features. Frankly, putting
an economizer on an air handler is almost no cost. It involves a connection
between the air handler and outside air and a mixing chamber. Youre talking
something thats clearly less than $1 million to do that.
Another things weve done is the containment. The cost to do that is 1 or 2
percent of the project cost so, again, its a no-brainer to use the pressure control,
limit your fan horsepower, et cetera. It wasnt anything where we modeled a lot.
We looked some different things up front but we were also trying to build a
building pretty quick. We interviewed architects and engineers in April of 2007
and delivered a building in December of 2008 so we didnt have a lot of time to
say Were going to explore or model A, B, and C. At the same time, we had the
prototype and we had a clear vision of what we were going to do.
which tends to be more conservative. I think just generally based on what Ive
The Art of the Data Center
seen talking to people, the more mature cultures look at this and say I get it,
but I would never be able to make this happen versus younger West Coast-type
companies where the early adopters they tend to be doing these sorts of things.
If someone wanted to fully load one of your 52U cabinets, whats the
15: NetApp
greatest power density you can support at a given individual cabinet
location?
Heres what I always tell people. You look in the cold room, through the door,
theres 720 kW of power. You can use it however you want in that room of up to
60 cabinets, you just cant have more than 42 kW in a particular cabinet.
The reason we can do that is, if you take a bathtub and you stick your arm into
it youre not going to change the water level in that bathtub as much as you try.
You might make some waves. If you put a cabinet in there with 42 kW of power
and its able to move the air through that cabinet the pressure in that room
immediately backfills. You cant change the pressure in the room, its physically
impossible to do it.
Our only limitation with this pressure control idea is based on power distribution.
The reason we get the 42 kW is every cabinet in this Data Center has two 21 kW
power strips in it.
Work closely with your internal customers and pick the right partners. In our
case I think having partners that were doing the design for us be local was really
important, versus somebody that had to fly in and it would have been a hardship
to get together. Particularly in the beginning youre doing a lot of brainstorming
in the design concept phase. Picking the right team and having a good
partnership internally with the user is really critical.
Figure 15-5
A walkway between
two server rows in
NetApps Research
Triangle Park Data
Center.
15: NetApp
297
Figure 16-1 Syracuse Universitys Data Center is both a production server environment and a
research facility. Images provided courtesy of Syracuse University.
Chapter 16
Syracuse University
Essential Details
Organization: Syracuse University
Location: Syracuse, New York
Online: December 2009
Notable features: Microturbines powered by natural gas. A trigeneration system that produces cooling,
heating, and power for the Data Center and nearby building. Use of direct current and alternating current
power. Closed-loop water cooling system using rear-door heat exchangers and sidecar heat exchangers.
LEED-Silver certified.
Time to design and build: 14 months
Size: 12,000 sq. ft. (1,114.8 sq. m) total, with 6,000 sq. ft. (557.4 sq. m) of hosting space.
Power: Two redundant 750 kW feeds; IT usable capacity of 600 kW
Tier: II
Cabinet locations: 70
Power Density: 8 kW average per cabinet, 35 kW maximum
Infrastructure Delivery: Structured cabling above cabinets. Electrical conduits, piping for liquid
cooling system, and air cooling delivered under a 36-in. (91.4 cm.) deep raised floor.
Structural loading: 450 lb. per sq. ft. (2197 kg per sq. m)
Fire suppression system: Novec 1230
Most modern server environments feature their share
Background
The Interview
It really drives a lot of the analysis and design choices we made here, to say
Can you give me an example of some of the things that are being
measured with this Data Center?
I say tongue-in-cheek that we measure everything. Thats a little bit of an
overstatement, but we have a lot of instrumentation. If you look on the power
distribution side, all the way from the grid connection or the turbines down
through that power distribution chain to the outlets in the racks, we can measure
power consumption. For the servers that were installing, the majority of them,
you can also ask the server How much power do you consume? So, you can
look all along that chain at what the performance of the system is. Similarly, in
the cooling space its heavily instrumented to let us know how the cooling system
is performing. Its really this notion of collecting large quantities of data about
performance of different portions of the system. The piece were working on now
is basically how do we take all this data, crunch it, and use it for both operational
purposes and also for research ones.
LEED Certification
Leadership in Energy and Environmental Design (LEED) is a rating system for the design, construction, and
operation of green buildings. Facilities can achieve ratings of certified, silver, gold, or platinum based on
criteria within six categories: sustainable sites, water efficiency, energy and atmosphere, materials and
resources, indoor environmental quality, and innovation and design.
302
For that research portion, once you have that data and youve been
The Art of the Data Center
able to do with it what you want, whats the intent? Is this something
that is to be shared with people who you might host in the facility? Is
it to be shared with the public so that the Data Center industry as a
whole can learn from it?
Certainly thats our hope, that we will share what we learn in various ways.
One of our traditional outlets, of course, is academic publishing. So we hope
to produce articles from our researchers and graduate students talking about
various ways to be efficient in the Data Center. One of the drives there is to
understandand this is an issue with construction of Data Centers as well as
buildings in general(why) designs dont often perform exactly the way that they
say they will. We really want to be in position to help refine those models so that
theyre more sophisticated in terms of what outcomes you get based on particular
design inputs.
Figure 16-2
Your facility doesnt neatly fit into the traditional tier classification
system for Data Centers. What drove your decisions regarding which
physical infrastructure components to have redundant?
Thats a great question. One of the things we did in constructing this Data
Center was to not necessarily be driven by rules of thumb or best practice. And
I dont mean to say that those things are necessarily bad. But we were really
looking to try to optimize everything that we could in the design of this facility.
When we looked at where to build in redundancy it was a risk-based approach.
What components do we think are likely to fail? We looked at service history for
microturbines, for example.
One of my big pieces in Data Center design is, I need it to be flexible. So, if
The Art of the Data Center
Figure 16-3
The American College and University Presidents Climate Commitment is an initiative to make higher education
Note
campuses sustainable and reduce their greenhouse gas emissions. Syracuse University was one of 152 charter
signatories of the ACUPCC in 2007. As part of the initiative, the University in 2009 issued a Climate Action Plan
that includes the goal of becoming carbon neutral by 2040.
306
If youre running a coal-fired electrical plant, youre sending a lot of heat out
The Art of the Data Center
the stack as waste, essentially, where in our case were actually able to use that
byproduct of the generation process either to provide cooling for the Data
Center or to provide heat for the adjacent facility. So we actually get a better full
utilization of the energy than we would otherwise and it makes that cost picture
look different.
Figure 16-4 Electrical switchgear for Syracuse Universitys Data Center routes power from
turbines to hardware, from turbines to backup batteries to keep them fully charged,
and from batteries to turbines to restart any that are idle.
307
If someone was building a Data Center facility and they were looking
When outside temperatures are low enough, you can leverage outside
The Art of the Data Center
cooling for your needs. How many days per year do you anticipate
being able to use it?
We anticipate 120 days per year of cooling will be possible from using heat
exchangers with outside air.
What operational temperature, at the server inlet, are you aiming for in
this Data Center?
Its interesting that you ask that. One of the analysis tasks for us is to look at
what is the optimal inlet temperature. There are these interesting issues that
happen. There has been a lot of work to say we should be raising Data Center
floor temperatures. The consensus is that thats a good thing and I dont intend
necessarily to argue with that consensus. But, with the way that were doing
cooling it isnt necessarily clear what the optimal point will be on the curve. You
start to look at what the efficiency profile of your air conditioning units is. In a
traditional Data Center you have to look at your air handlers, whether theyre
CRACs (computer room air conditioners) or CRAHs (computer room air
handlers), and the energy youre spending blowing air around. That optimization
can look a little different when youre using rear door or sidecar heat exchangers
because it might make sense to maybe run it a little higher, it might make sense
to run it a little lower. You really have to analyze this to know the answer to that
question.
I cant quote any of these for a variety of reasons right now, but I know that
when weve done some of the modeling work some of the results have been
counterintuitive. So we want to make sure that before we set that design point
that were getting the optimal efficiency from it.
309
You have hinted at the liquid cooling solution that youre using in the
Figure 16-5 Hardware within Syracuse Universitys Data Center. Shown are an IBM storage array, a
mainframe that uses direct current power and a supercomputer with liquid cooling to
the chip. The liquid cooling is integrated with the buildings chilled water system. The
three full-height rack doors on the left are rear-door heat exchangers.
310
In the Data Center industry the general take on liquid cooling is that
The Art of the Data Center
its more efficient than an air system but that it has a higher capital
cost. Theres also a lingering concern about water in the Data Center.
Many folks have worked hard to keep liquid off of their floor and are
hesitant to bring it back to the cabinet. Did you have any concerns
along those lines?
Well, Id be lying if I said I didnt have some concerns at least when we were
starting down this road. But when you actually look at the systems that were
putting in place here, the truth is that there really is not a significant increase
in the risks in operating a Data Center. If you look at our rear door exchangers,
for example, were doing the chilled water distribution under floor and the
connections for the rear door and the sidecar exchangers are both down at the
bottom. So, if you look at where we were likely to have a leak, its all down below
the place where we have equipment.
You go back and actuallyagain, IBM is a great resource in thiswe did this
for a long time and the record of these liquid cooling systems is actually pretty
good. Were there zero incidents of problem? No. But frankly we have water issues
in a number of places. One of the things that was interesting to me as we were
beginning this design process was the number of Data Centers I visited that have
sprinkler systems in them. The basic thing is, look, were very unlikely to have a
fire here and they have dry pipe systems, but when we looked at the cost of other
fire suppression, we decided this is the better risk/reward ratio.
So, is it a concern? Yes. But I can tell you that if its up to me I wont build
another raised floor Data Center. I will do distribution overhead just to avoid
the capital cost and other issues associated with a raised floor. And that includes
putting chilled water overhead. If its done right I dont think its any significant
risk profile we cant deal with.
311
Do you have the option to provide AC power for the Data Center
The Art of the Data Center
as well?
Oh, yeah. The distribution is that we have 500 kW of AC power and about 100
kW of DC power available to the floor. We cant get everything that we want to
run in our Data Center DC power-equipped. So, the majority of the equipment
in there we anticipate being AC-powered for at least the next few years.
Were there any special steps that you needed to take because you
have both types of power going in to the facility?
A little bit. You want to make sure its clear to your electrician whats what. In
a typical Data Center you can look at a cable and know what it is. But in ours,
given that we have both AC and DC power distribution, those folks have to be
oriented. The distribution systems are a little different and there are some things
that you have to know, but I was actually pleasantly surprised both in terms of
the folks that worked with us on the construction but also our own (Syracuse
University) electricians that this is not as big a deal in terms of a transition.
As a side note, that is also true because in the larger mechanical systems for
buildings theyre sort of blurring these lines anyway in terms of the large blowers
and pumps and things you use as part of normal infrastructure. So, these
electricians have to be pretty sophisticated folks generally. This is not as much of a
curveball as you might think.
What lessons did you learn during the design and construction of
the facility?
One of the things that I learned is, boy do you have to have the right team to
do this kind of project. There are lots of architectural and engineering firms,
Data Center design firms. You can hire them and get a great conventional Data
Center, and even a reasonably efficient conventional Data Center, but if you want
to color outside the lines, you really have to have a team thats comfortable with
coloring outside the lines. That includes the CIO who has to be willing look at
things and say Well, I know no one else has done this, but Im looking at the way
that this is arranged and the nature of the components and the track record and
whats proven, et cetera. And I am willing to certify, yes I will run my production
systems on that.
313
Another piece I often mention, and this is particularly true for larger companies
Were there any challenges or surprises that came up that you didnt
expect when you started?
I was actually surprised at essentially how straightforward it was to get all of these
folks working together on something that was so out of the box. The people were
great, but also it was really a willingness to look at all of the different components.
Most of what we have in this Data Center is stuff thats off the shelf from
somewhere. There are a few things in there that weve done that are unique and
you really cant find anywhere else in the world. The particular microturbines that
we have out there now I believe they were first of a kind and so far are only of a
kind, but theyre going to be an off-the-shelf item now from Capstone (Turbine
Corp.). Outside of those things, this was Lets look out there, find all these things
and lets put them together in a way no one else has. And that was maybe easier
than I thought it would be at the outset.
This is a relatively young facility, but if you could go back and start
the design all over again what, if anything, would you do differently?
Let me answer that question two different ways. If I look at doing it with the
same missionin other words I want it to be a research facility as well as a
production Data Centerprobably not very much. We really need all of the
different aspects of what we have there to satisfy both of those missions.
If you said Hey, Chris, I want you to start again. I want you to build something
thats equivalently efficient but its really going to be a production Data Center
I would look seriously at dropping the raised floor. There probably are some
other minor things we would look at. One of the things that we did do relatively
314
quickly after we constructed it was we built some caged space inside the Data
The Art of the Data Center
Figure 16-6 Two 150-ton absorption chillers help cool Syracuse Universitys Data Center.
315
with one server manufacturer that I wont name and they said, basically, Look,
if I cant sell a million units I cant make a business case for this. Theres going
to need to be some instigation in the market that really drives an efficiency gain
here and that may take 5 or 10 years. When people ask Should I put DC in? I
say generally if you have equipment that you know you can run on DC and its
sufficient to pay for itself at this point yes I would put in DC. If you dont have
those characteristics in what youre doing, I would make provisions so that if
youre planning on having a facility life that extends beyond 10 years that youre
able to install it and operate it in that facility but it isnt something that I say
generically everyone should be putting in now.
You have the goal that at some point this Data Center will be able to
flexibly provision both its application workload and its cooling. Move
the IT load, move the facility resource to be able to support it. How do
you envision being able to accomplish this?
Thats a great question. There really are two dimensions to that. One is the IT
dimension. How do you distribute workload across the compute resources and
storage resources and network resources that you have in your Data Center.
For us, the big play here is in the virtualization space. Were doing experiments
with VMwares vSphere product. They can automatically load balance, they can
consolidate, they can actually turn physical servers on and off to add capacity or
subtract capacity based on load. So, on the IT side its really leveraging the tools
that we have there to provide the right mix of redundancy and performance and
energy efficiency.
On the Facilities side the challenge and the opportunity is to, number one,
factor in the Facilities parameters in to those choices so that you can say Well, it
would be best from a Facilities systems perspective if we could consolidate load
geographically within the Data Center as well as just picking 10 different servers
that are in 10 different racks as the destinations for it. The other thing is, its nice
to be able to tell the Facilities systems what youre doing from a consumption
317
perspective so that they can produce that optimally. And Im not claiming that we
Do you have any final thoughts that you would like to share regarding
this facility or Data Center design practices in general?
Probably the parting thought that I would share is a challenge to CIOs, to be
willing to try new things and, with appropriate due diligence, take some risks to
build facilities that are new and innovative rather than sticking to whats known
to be safe.
Figure 16-7 A closeup of the Syracuse University Data Center buildings outer wall.
Figure 16-8 Hot exhaust (585 degrees Fahrenheit/307 degrees Celsius) from microturbines flow
through heat exchangers, shown here, producing hot water that is then piped to an
adjacent building to heat it.
319
Figure 16-10 An IBM Rear Door Heat eXchanger cooling door removes heat from the hardware
within a cabinet inside Syracuse Universitys Data Center.
Figure 17-1 Rooftop satellite dishes provide backup connectivity for Terremarks NAP (Network
Access Point) of the Americas building. Images provided courtesy of Terremark.
Chapter 17
Terremark
Essential Details
ORGANIZATION: Terremark
Location: Miami, Florida
Online: June 2001
Notable features: One of the largest single-building Data Centers on the planet. Capable of withstanding
Category 5 hurricane winds. Lightning prevention system that defends against strikes by discharging static
electricity flares.
Time to design and build: Ongoing
Size: 750,000 sq. ft. (66,677.3 sq. m) total, with 600,000 sq. ft. (55,741.8 sq. m) of hosting space
anticipated upon buildout.
Power: Designed for 70MW upon buildout
Tier: III
Cabinet locations: Varies by customer configuration
Power Density: Varies by customer configuration
Infrastructure Delivery: Power and cooling are delivered under a raised floor, structured cabling is
provided overhead.
Structural loading: 180 to 280 lb. per sq. ft. (878.8 to 1,367.1 kg per sq. m)
Fire suppression system: Pre-action dry pipe
Terremarks flagship Data Center is a testimony to
Background
thinking big.
When the massive co-location facility known as the
NAP (Network Access Point) of the Americas was
proposed for construction in 2000, Terremark was a
real estate development company with no Data Center
design or operations experience. The dot-com bubble
burst early that year, making the outlook for Internet-
related companiesand co-location centers that rely
upon them as customersgrim.
Despite that, Terremark moved in to the business in a big way, constructing one of the largest Data Centers
on the planet. (The company now has about 50 facilities.) At 750,000 sq. ft. (66,677.3 sq. m) and with an
anticipated 70 MW of electrical capacity upon buildout, the six-story buildings capacity dwarfs what some high
tech companies have for their entire global Data Center portfolio.
The immense structure features Tier III standby infrastructure, server cabinets up to 10 ft. (3 m) tall, and
almost unlimited power densityas of this writing, two U.S. government agencies are in discussions with
Terremark to host a hardware deployment requiring 1,000 watts per sq. ft. (10,763.9 watts per sq. m) of power
and cooling.
Although the scale of the Data Center seems daunting, the abundance of its physical infrastructure elements
space, power, and coolingare actually its greatest strength, providing design and operational flexibility not
usually available in smaller installations.
Ben Stewart, senior vice president of Facilities Engineering for Terremark, discusses why bigger has proven to be
better for the NAP of the Americas.
The Interview
What drove the decision to locate this facility in Miami?
If you look on the Eastern seaboard of the U.S. all the transoceanic cables come
in to New York or Florida. Most of the Florida cable systems service Central and
South America, though we do have Columbus 3 which comes over from the
Mediterranean into the South Florida coast.
All those cable heads come into south Florida and they had to find a place
where they could peer their traffic. They were buying tail circuits up to Ashburn,
323
Chicago, and other major peering centers to conduct that. So, a consortium
17: Terremark
pooled their money to build a facility in Miami where all these cable heads turn
up in to and actually staged their traffic right here in Miami. Terremark, a real
estate development company at the time, won the contract to build this facility
and then somewhere through that process decided that it was going to shift its
business model and become a designer, operator, and owner of these kind of
facilities around the world.
So thats really the genesis of the facility. It was the cable heads coming in to
South Florida and needing a place to terminate in to and exchange their traffic
with all of the terrestrial carriers as well. We now have 172 carriers inside this
building who are exchanging traffic with each other.
We dont meter our power, we charge by the whip. And the more power you buy
The Art of the Data Center
the more cooling youre obviously paying for with that. So its really up to us to
figure out how were going to deliver that cooling.
Obviously each floor is designed and built to some power density. For instance
our second floor, which is the first floor we built more than 10 years ago, it was
designed and built to 80 watts per sq. ft. (861.1 watts per sq. m). But thats over
roughly 80,000/90,000 sq. ft. (7,432.2/8,361.2 sq. m) of white space so we can
move that cooling around very easily. And in the early days we had a lot of carriers
coming in at about 30 to 35 watts per sq. ft. (322.9 to 376.7 watts per sq. m).
On the other side of our floor weve got a customer pulling down about 300
to 400 watts per sq. ft. (3,229.2 to 4,305.6 watts per sq. m). And we have no
problem delivering that because thats just static pressure under the floor. You
throw more tiles down and you get the air in there that they need to cool those
servers.
With such large floors we dont dictate that to our customers. Were in the
business of making this space work for any customer that could ever possibly
want to come in here and weve built and designed that sort of flexibility into
our floors.
Whats the most extreme request that you have had to accommodate?
A customer came in with some mainframe type of equipment, some real high
power type stuff and theyre running probably in the 300 to 400 watts per sq. ft.
(3,229.2 to 4,305.6 watts per sq. m) range in a very small area. Thats probably
our heaviest one in this facility.
How does it influence your Data Center design, knowing that youre
going to have to accommodate a range of client needs? How do you
ensure that you have flexibility within your infrastructure?
On the cooling side, which is usually the more difficult side because you can
distribute power from other floors and things, its really the raised floor. The raised
floor and the ability to move our cooling around by moving perforated tiles.
I read a lot about people talking especially on the enterprise side going away
from raised floors and building directly on the slab and ducting the cold air and
325
ducting the hot air out. Thats a great idea, energy efficiency-wise thats probably
17: Terremark
one of the better ways to go. But when you do a build-it-and-they-will-come
model where youve really got to have the flexibility and you dont know who is
coming or how much theyre going to need and youre designing their layouts on
the fly, that raised floor just provides an enormous amount of flexibility.
Figure 17-2 An unoccupied hosting area within Terremarks NAP of the Americas facility.
How long did it take to design and build the NAP of the Americas
facility?
Were really designing it on an ongoing basis.
When the site was originally built we only built out the second floor, so the design
basis and the build was only to power and cool the second floor. As we go from
floor to floor we continue to modify that design basis as we learn things and try new
things on an energy efficiency basis. Each floor is just a little bit different as we go.
326
The time period to design this, they started construction in August of 2000
The Art of the Data Center
and opened the doors operationally in June of 2001 so it was less than a year to
put this entire structure up. It was built on a parking lot so it is a purpose-built
building. But the design was only about 6 months and design was ongoing as
they constructed the facility, to include adding floors.
You mention changing your design and build approach based on what
you have learned. What are some of the things in recent buildouts that
you didnt do originally?
The second most recent one is that we put in a drop ceiling so we have a hot
air plenum as well. Now not only do we move our cold aisles by moving our
perforated tiles we move our hot aisles by moving egg crate thats hanging in the
drop ceiling above. So even on a better basis were separating our hot air from
our cold air and then taking the CRACs (computer room air conditioners) and
putting them straight up into the hot air plenum. Were sucking the hot air out of
the hot aisle instead of letting it just naturally convect up in the overhead and pull
off to the perimeter coolers.
I say two times ago because we improved on that a little bit more. That proved
to be very difficult for our cable management systems because, again, we build it
out in advance. Having that drop ceiling in there and trying to put Allthread up
into the overhead to drop trapezes and ladder rack and stuff down to run cable
management to where our customers want to be proved to be pretty challenging.
We modified that a little bit by using something called Unistrut. We put in a big
green grid overhead of all of our floors where we hang our lighting and our cable
management and everything. When we did the drop ceiling that conflicted with
that so we got the Armstrong drop ceiling folks together with our Unistrut folks
and now we actually build the drop ceiling into that Unistrut so it doesnt conflict
with our cable management in any way, shape, or form. So we can still have the
benefit of the hot air plenum without the operational complexity of trying to
calculate your cable management in advance so it doesnt conflict with the drop
ceilingits all one now.
327
When I interview most Data Center designers I typically ask what they
17: Terremark
would do differently if they had an opportunity to go back and design
their facility again. That doesnt seem applicable here because youre
getting to design sections of the facility on an ongoing basis.
Excellent point. Thats exactly right, because as we do each floor we learn a little
bit more.
We have a facility up in Culpepper, Virginia, where were building individual
50,000 sq. ft. (4,645.2 sq. m) pods on 30 acres (12.1 hectares)in fact, we
just bought another 30 acres (12.1 hectares) so its going to be a 60 acre (24.3
hectares) piece of property. The original design on the original 30 was to build
five 50s. So A was our first one, we built B and C, were in the process of D. I tell
people that by the time we build E were going to want to burn A down.
Figure 17-3 Clients use cabinets up to 10 feet (3 meters) tall in Terremarks Data Center.
328
Are there any early installations where you cant wait for them to
The Art of the Data Center
become vacant so that you can go back and re-design them because
you know much more now?
On our second floor, back when we were first just trying to stay alive and put
customers on the floor, we kind of allowed customers to go on the floor wherever
they wanted, however they wanted, oriented any way they wanted. So our cold
aisle/hot aisle orientation is not optimal in the way that we actually had some
customers come in and orient their servers. Theyre sucking air in from the hot
aisle and blowing hot air out into the cold aisle. Weve had to come in and take
them down and turn them around because we just couldnt tolerate that.
I would love to distribute the carriers around a little more. When we built out
the second floor we built from the west to the east so on one side of the building
weve got a whole bunch of 35 watts per sq. ft. (376.7 watts per sq. m) customers
and on the other side of the floor weve got a whole bunch of 400 watts per sq.
ft. (4,305.6 watts per sq. m) customers. Kind of would liked to have mixed that
up a little better, but when youre in a constrained revenue environment there are
certain things you have to do to just stay alive.
That makes sense. Even for companies with in-house Data Centers if
they undergo huge growth surges it requires discipline and foresight to
closely manage installations and not allow the drive to bring hardware
online rapidly to take precedence over everything else.
Youre absolutely right. Engineering and Operations, one group wants to stick to
the rules and the other ones say they cant stick to the rules. Its a constant battle.
Its a yin and yang sort of thing, but the business model will vet out what the
right answer is.
329
17: Terremark
much larger overall than most other Data Centers. Did any challenges
come with designing and building a server environment on such an
immense scale?
To be honest with you I would say the oppositeit made it a bit easier. Because
we have so much power and so much cooling that we can spread around to so
many different places, that kind of flexibility has made it easy for us to bring
customers in and when they wanted something very unique show them the
unique things we could do for them. A lot of the other places they have been
to take a look (they are told), Here are your three choices, which one works for
you?
Weve got all this white space and all this pressure under the floor and all this
power distribution capability that we havent distributed yet. You tell me what
you want and we can work with you. Having so much capacity in terms of space,
power, and cooling to move aroundthe built-in flexibility to do thatits
proven to be pretty powerful. If we just had a bunch of 2,000 sq. ft. (185.8 sq. m)
rooms that would be a lot more difficult to do.
money, wasted capital, wasted maintenance because the floor will not tolerate
The Art of the Data Center
Your facility has satellite dishes on the roof that provide backup
connectivity in the event a problem occurs with your fiber cabling.
What can you tell me about them?
The dishes on the roof, there are two 16.4 m. (53.8 ft.) C-band dishes and then
the smaller one that looks more like a geodesic dome thats an 11 m. (36.1 ft.)
Ku-band dish. Those are all pointed to commercial satellites and on the other side
wherever the customer isI really cant talk too much about whos actually using
those servicesthey just put a small VSAT (very small aperature terminal satellite
communication system) up on the side of their building and if for any reason they
were to lose their terrestrial connectivitya fiber system went down, a central
office caught fire, or somethingwe can just switch over to the satellite modems.
Its obviously not at the same bandwidth but at a lower bandwidth reestablish that
connection to the VSAT terminal sitting on the side of their building going across
that commercial satellite.
Was this something you had in mind during the initial construction of
17: Terremark
the facility?
No. Absolutely not. Weve learned so much from this facility. For instance, when
it was first built we never envisioned in a million years that the U.S. government
would want to be inside here, however in retrospect the Southern Command is
here (so) it makes perfectly good sense for them to be here.
The Southern Commands area of operations is Central and South America and
all those cable systems come into this building. So when theyre running it off
down there sometimes they need to hook up some terrestrial circuits in a real
hurry. If you want to hook up to something in a real hurrywhat they call zero
mile connectivityall the carriers are right here in the building. We put in a
cross-connect and youre up as long as the tail circuit is on the other end, youre
up and running right away. Were turning up circuits for the Southern Command
in hours that used to take them 6 to 9 months.
We never envisioned that before. Satellite dishes on the roof? Absolutely not. In
fact we had mapped out the entire roof because we dont have a lot of equipment
yard here. Everything has got to go on the roof. We had mapped out the entire
roof for 7,000 tons of chiller up there, how we would lay it out and how we
would grow into it.
When the need for the satellite dishes came around we had to consume an awful
lot of that real estate on the roof for the dishes. That was not planned. However,
all the dunnage was built onto the roof of the chillers and as luck would have
it we were able to saddle off the weight to the columns straight down to the
bedrock. We were able to pull it off. Again, that flexibility for these customers
that wanted it here. And now weve gone to a more vertical style chiller to account
for the lost square footage on the roof.
332
The Art of the Data Center
Figure 17-4 Four inch (10.2 centimeter) conduits turn up into Terremarks building from an outside
underground vault. In the conduits are innerduct with fiber optic cable and pull strings
protruding from the innerduct.
333
17: Terremark
customers, but do you feel that the design elements overall are
universal and can be installed in any Data Center or are they point
solutions that are only appropriate in certain circumstances?
There are point solutions, but these are things anybody can use anywhere and
they are being used anywhere and if they had floors this size they would be doing
it this way. Like I say, it gives you that kind of flexibility.
When I refer to point solutions, we do have some customers that are in 2,000 sq.
ft. (185.8 sq. m) suites. We do have some of those here, which incidentally were
originally designed to be the battery rooms and instead we went with rotary UPS
and we opened them up to be sold as private suites.
Sometimes it gets so hot in those rooms that if you put downflow coolers in
there you wouldnt be able to put any cabinets in because they were going to
consume so much power. So we went to cold aisle containment and in-row
cooler technology inside those rooms. That would be a point solution example of
how we accommodated high power density in a very small room, where we just
couldnt get enough air handlers in there to handle it directly.
With all of the retrofits and expansions that you have done along the
way, even though youre only working with one facility it seems like
you have actually built multiple Data Centers.
Youre absolutely right.
servers with big heat syncs on them, really cool stuff. With a co-location center,
The Art of the Data Center
what are we going to tell our customers, We love to have you come here, well sell
you some square footage but youve got to build your own servers. Weve got to
meet everybodys needs.
If we had a customer come to us and say I need about 30,000 sq. ft. (2,787.1 sq.
m). I need you to fence that off for me, and I need you to help me come up with
a really, really energy efficient solution just for my 30,000 sq. ft. (2,787.1 sq. m),
what a ball that would be, because now all that flexibility isnt required anymore.
We can do anything we want inside that space because we know the hardware
thats going in there, all of the variables that we have to consider on the outside
somehow suddenly become fixed constants and we can design to that.
I love it when people say Ive seen Data Centers, theyre all the same. I disagree.
Data Centers are like fingerprints, theyre all so very different. You can class them
and categorize them but even within their categories they change theyre so very
different. In a co-location environment its very simple, were in the business of
selling space. We want to consume as little of it as possible yet keep it as flexible
as possible(those are) almost diametrically opposed, but nonetheless that is our
goal when we go out and look at white space to develop and grow. Use as little of
it as you have to for your own stuff but keep it as flexible as you possibly can. And
that doesnt work for an enterprise. That would be ridiculous.
17: Terremark
by physical space for decades, until the 2000s when high-density
hardware made capacity about power and cooling for most companies.
But youre saying that for your business Data Center capacity still
comes down to physical space.
Correct. Well, well usually run out of power and cooling before well run out
of space but we are very, very careful about how we carve up our space for our
customers.
Im sure youve read about our cloud environment. Were very virtualized and
virtualizing increases your power density. Youre putting the same amount of
compute into a smaller area. It doesnt really save you a lot in the way of energy,
although some people are quoting about 15 percent of power savings by piling
it all over into one corner. But by piling it over in one corner just complicates
your cooling solution. Yet thats a good thing to do because it frees up more white
space that you can build more power and cooling and customers in to. Its a
dynamic.
What we found is the cloud computing environment, which is a very small part
The Art of the Data Center
You are, by necessity and design, getting to use a lot of different Data
Center technologies in your facility.
This is a playground.
Figure 17-5 An enclosed hosting area within Terremarks NAP of the Americas facility.
337
Anything out there that you havent had a chance to employ that
17: Terremark
youre hoping to?
I keep reading about more and more of these places like the Yahoo! Data Center
in Lockport or Facebooks Data Center in Prineville using this evaporative cooling
where theyre actually building a center with no chillers. They arent installing any
chillers at all. We arent building up in those latitudes that allow us to do that, but
we do use waterside free cooling, airside free cooling. Obviously no free cooling
here in Miami but some of our other sites where free cooling makes some sense
were able to do that.
But we really havent been able to get into the evaporative cooling part of the
world yet. Its something were always looking at because its obviously going to
save us a great deal of money in terms of utility, a lot of opex savings. The other
side of it too, though, we dont evaporate any water at all. All of our chillers,
everything is air cooled. We see that as a risk. Right now all of the focus is on
Data Centers and the amount of energy they consume. Youre already starting to
read some things about how much water theyre consuming and where their water
sources are and where their backup water sources are. Its kind of nice when we
have a customer come to us and grill us about those sorts of things and we get to
the water piece and they ask Whats your backup water supply? We dont need
any water. We dont use water. We dont evaporate water. We have no evaporative
cooling.
So, right now we dont have that anywhere. Yet I keep reading about the benefits
of it and places with no chillers. How exciting is that? Would love to explore that
a little bit further, but currently were just not in any area dry enough to allow us
to do that.
Any final thoughts that you would offer someone approaching a Data
Center project? Any lessons learned as you have solved issues for
various customers needing Data Center capabilities?
When we solve them for our customers usually theyre coming to us because
theyre tired of trying to do it for themselves. But I do talk to and deal with other
companies and people in professional associations, people who are trying to still
roll their own, trying to do it for themselves.
338
My only advice to them is, Data Centers have a lot of moving parts. People think
The Art of the Data Center
theyre very simple, theyre very straightforward and there really isnt that much
to it. Even if you just break it down electrically and mechanically, there are a lot
of moving parts, a lot of options. Theres no one model thats perfect for anybody.
There are tradeoffs, there are pros and cons. They really need to partner with
somebody else who has got that kind of experience or take the time to develop
that experience on their own through their own study to really make those kinds
of decisions for themselves. I have seen so many Data Centers designed for an
enterprise that did not take into account anything to do with their business model
or what it is they were trying to do. And they were just miserably disappointed
with the product that they got because it doesnt support their business because
nobody took that into account.
Even some of your large companies out there will just go to a major A&E firm
and say I need a 100,000 sq. ft. (9,290.3 sq. m) Data Center. The firm says Ok,
well what for? I dont know, I just need it. And they get a Data Center and its
built to Tier III or something to that effect and they get in there and say, This
isnt working for us at all. Its not shaped right, its not powered right, it doesnt
have the flexibility. Yet all those things seem to come out afterwards, and thats
because the people doing the buying and the people doing the building didnt
shake hands very well.
There are just a lot of moving parts, a lot of things to consider. I love it when
someone comes to me and says Ah, a Data Center is a Data Center. Yeah, there
are some differences but in the end theyre all the same. I just smile and say My
God I hope you never build one.
339
17: Terremark
Figure 17-6 Distributing cooling beneath a raised floor and strategically placing perforated floor tiles
gives Terremark maximum flexibility to support customer deployments.
Figure 18-1 The Yahoo! Computing Coop in Lockport, New York employs no chillers or mechanical
refrigeration. Images provided courtesy of Yahoo!.
Chapter 18
Yahoo!
Essential Details
ORGANIZATION: Yahoo!
Location: Lockport, New York
Online: September 2010
Notable features: No chillers or mechanical refrigeration. Airside economization and evaporative cooling
system. Hot aisle containment. Rotary UPS. Shared space for electrical and IT infrastructure.
Time to design and build: 9 months from ground-breaking to commissioning
Size: 190,000 sq. ft. (17,651.6 sq. m) total
Power: 20 MW
Tier: Varies
Cabinet locations: 2,800
Power Density: Undisclosed
Infrastructure Delivery: Power, cabling, and cooling infrastructure are delivered overhead
Structural loading: Undisclosed
Fire suppression system: Dual interlock, pre-action dry pipe
Twenty miles east of Niagara Falls sits a cluster of pre-fabricated
Background
The Interview
What factors prompted you to locate the Data Center in western
New York?
Paul: One of the criteria the site selection committee looked at was the weather.
To implement this design requires a certain weather profile. That was the number
one driver in being able to deploy this technology. The other was proximity to
renewable energy and hydro power. That was a major factor. And a location where
all the other criteria that would go into a Data Center site could be met.
Chris: Access to power that was both green and affordable was a key component.
Some other factors were network connectivity from multiple providers and
reasonably priced land. Tax benefits and the first-rate workforce present in the
state of New York also played an important role.
343
Thats interesting. You didnt choose the site and then come up with
18: Yahoo!
the design. You planned to use this design and went looking for a
place where you could leverage the climate to accomplish it.
Chris: Correct.
This Data Center is best known for how its cooled. Can you walk me
through how thats done?
Paul: Essentially theres an air handling system for the computer room segments
that takes in outside air, filters the air, and then there are fans that move it into
the room. That air handler makes use of return air from the IT equipment to mix
with outside air to achieve the set point. Thats all done through building controls
and automation with outside air and return air damper systems.
The only means of cooling that we use is the evaporative cooling system, which
we deploy when we exceed our desired set point. The evaporative cooling
system is a passive cooling system that uses tap waterno artificial means of
pressurization or anything like thatand evaporative media to cool the air thats
coming in through that air handler.
That creates the cold aisle temperature or inlet temperature to the racks that we
desire, and then we use hot aisle containment. Hot aisle containment captures the
heat off of the back of the cabinets and discharges it through the top of what you
may have seen as the chicken coop in pictures of the site. That air is not recycled
unless we want to use it to warm the air in the cooler months to reach our set
point.
I understand you were very precise with the cooling design, down to
the point of how the buildings are oriented so as to take advantage of
how the wind blows across the property.
Nick: The reasoning for the buildings being tilted like they are was actually two-
fold. One was position on the land that made the most effective use of the land
itself. Additionally, you didnt want a building that dumped hot air onto itself.
Positioning the building in a certain way would allow the prevailing winds to
push the exhaust heat away from the building. When you placed the building you
344
wanted to make sure you didnt locate it where the prevailing winds would push
The Art of the Data Center
the hot exhaust into the intake. So the alignment of the building was based off
prevailing winds but it wasnt a major design factor, but rather a positioning factor.
Figure 18-2
Yahoo! positioned
its Data Center
buildings to
maximize space on
the property and
to ensure exhaust
heat was blown
away from the
structures.
Ive seen both hot and cold aisle containment used in Data Centers.
What caused you to favor hot aisle containment?
Nick: As you go through the history of the change of the Data Center model
from more of an open floor plan to more of the hot and cold aisle configurations,
initially cold aisle configurations were set up because the distribution of the cold
air was via raised floor, under that floor to the front of the aisles. As soon as raised
floors went away, and we moved away from distribution of air under the floor tiles
to air from in front of the servers, it became possible to do hot aisle containment.
So, a big change that came from cold aisle containment to hot aisle containment
was because we were no longer distributing with a raised floor. That allowed you
to open the front of the servers up and not have to wonder where the air was
coming from specifically but wonder where its coming from fundamentally,
because as you open the room up the whole room became the cold aisle versus
just right in front of the cabinet. So, that was the reasoning behind transitioning
from cold aisle containment to hot aisle containment.
345
Was the decision to forgo a raised floor tied to the use of the
18: Yahoo!
cooling design?
Nick: The reasoning for removal of the raised floor was that it was not needed.
Designs that do not have under floor distribution of air can still use raised floor
we do that in some locations, for cabling and for infrastructure under the floor.
But you can do cabling above the cabinets, so in most cases raised floors arent
needed.
Paul: Its a huge savings on construction, too. Massive.
Nick: And structurally as well. You had to make all these cabinet positions be
structurally sound on a raised floor and its a lot easier to do it on slab.
Bob: And when youre building from a green field you actually have the option
of running it all in conduits under the slab, which you wouldnt be able to do in,
say, an existing building which might make it tougher to go with a non-raised
floor. You would be running so much overhead.
leveraging outside air, either per cabinet or on a watts per square foot
basis?
Nick: Id say no at this point. Because youre controlling the cold aisles, but
the (ultimate) control point is the hot aisle. So as long as your hot aisle doesnt
overheat then, yeah, you can deploy anything you want in there.
What was the inspiration for the Data Centers cooling design? I love
the thought of someone at Yahoo! sitting at their desk, looking out the
window and suddenly thinking of a chicken coop, but was it that easy?
Nick: Ill take you on a short journey of how we got there. We start with YTC
(Yahoo Thermal Cooling), which is our design regarding cold aisle encapsulation
with cooling coils. Basically, take the cooling coil out of a CRAC (computer room
air conditioner) unit, place it between the two rows of servers, and let the servers
move the air. That design was based off of getting rid of CRAC units because
CRAC units of course draw a lot of energy. They take up a lot of space and theyre
maintenance intensive.
That was to resolve an issue regarding CRAC units. When we went into the
YCC design the basis was basically like a circus tent. Why do we need all these
components that we have inside the Data Center? Why cant we just use the fresh
air thats outside? Utilizing different tests that we did by placing cabinets outside
and all these other tests that we completed prior to the mockup of the actual
design, we decided we didnt need all those components that currently were inside
a legacy Data Center. It was really trying to think outside the box and decide can
we do this? Well, we determined that we could.
We also determined that we didnt need all the control points, the stringent
humidity requirements, the stringent temperature requirements. They were no
longer needed and were a legacy from punch card days when keeping humidity
between 45 and 55 percent was required to keep punch cards from sticking.
That wasnt required anymore. As long as we keep a consistent temperature and
a tempered humidity we were going to be fine. And thats what drove us to this
design.
347
18: Yahoo!
Figure 18-3 Power, cooling, and cabling infrastructure are delivered overhead to cabinet locations in
Yahoos Lockport Data Center.
That drove the industry to a) recognize some great opportunities that were
The Art of the Data Center
How long did it take to design and build the first phase of the site?
Nick: We went from YTC design straight in to YCC. So, for the design length of
time is kind of a fluid number, because of the different types of testings that we
did complete. To go from a fundamental theory to an actual design was probably
six months and then youve got to work yourself in to engineering and all that
stuff to get an actual design.
We then went to a media which is like a swamp cooler media, versus the atomized
18: Yahoo!
air concept that saturated everything, and found that when we tested the swamp
cooler approach everything worked great. During that day we did two tests. One
was heat the building up really, really high, to about 104 degrees (40 Celsius),
and then allowed the moisture to cool the air. We additionally added humidity to
the air to see the fundamentals of what happened at the server level with elevated
humidity as well as elevated temperatures. We were able to get a full swing of data
from from one extreme to the next extreme.
Additionally, we tested the low temperature side, using recirculation. When you
put in cold air to a Data Center thats actually worse than putting in hot air,
because if you have a Data Center sitting in high temperatures and you blast it
with low temperature air youre going to break some stuff. So, the solder joints
will be sitting at high temperature and youll hit it with extremely cold air, the
solder joints will retract and the servers will break themselves.
So testing the high temperature as well as the low temperature as well as the
air distribution flow, in a controlled environment with data collection, were all
important.
Did you find a sweet spot? Conditions that were ideal to operate within
that you then wanted to try and find in nature?
Nick: Its not so much a temperature you create; its the consistency you create.
You can run a server at 90 degrees (32.2 Celsius) inlet if youd like, but you need
to do that continuously and consistently. Dont go up and down, up and down,
up and down. You can run a server at 50 degrees (10 Celsius) but dont do that
inconsistently. Its all about consistent temperatures is what really is the saver.
Paul: Like Nick was saying, controlling the rate of change is obviously a big
concern when youre talking about outside air. I described that recirculation
feature before where we can hold a set point. That rate of change is very easy for
us to control during most conditions throughout the year. And then when we
are subject to the rate of change that the outside air throws at us, its really not an
issue at that point, which would be more toward your summer months.
We also find that in western New York, for most of the year you can run at
however low a temperature you want in the computer rooms without having to
increase mechanical loads, which is kind of a unique feature of a building of this
design. Whereas a server might consume less power at lower inlet temperatures,
350
if the server fans are driving off of server inlet temperature, most Data Centers
The Art of the Data Center
Figure 18-4
A worker stands
inside the cupola
atop one of the
Yahoo! Computing
coops. Hardware
exhaust is
discharged out of
the building from
here.
failed. You cant just open a window in those kinds of Data Centers. But that is
18: Yahoo!
what we do herewe can open those windows.
If youre asking about availability perspective, thats one of those things that I
dont think you had to have this design criteria per se because it was inherent
within the reduction of that complexity.
Chris: Frequently reducing your complexity increases reliability; it also reduces
cost. That was one of the things here. Cost effectiveness and reliability were
priority one. I think people have this mental model that when youre talking
about energy efficiency or green, its going to be more expensive up front and
you really have to sweat the payback period. In some cases, especially with the
simplicity of design, it can be cheaper and faster to build.
Youve got a group of equipment associated with one system, associated with a
The Art of the Data Center
segment of the cooling system. It makes it very repeatable here. It allows you
to scale in a more controlled fashion, versus building out a massive segment of
power and then filling it up we can reduce the time needed to deploy a segment
of this thing.
Figure 18-5 A view from inside one of the Yahoo! Computing Coops contained hot aisles. 18: Yahoo!
354
This is a still a relatively young facility, but if you had to go back and
The Art of the Data Center
18: Yahoo!
goals? If youre trying to design a Data Center without a conventional
cooling system, that presumably requires buy-in from both IT
that owns the hardware and Facilities that maintains the physical
infrastructure.
Nick: Since 2005, we as an organization have run IT and Facilities under the
same VP. Theres no win-loss. A lot of our designs, going back to YTC designs and
also the cold and hot aisle configurations, come from a true partnership between
Facilities and IT. A facilities manager does the infrastructure and Site Operations
does the servers and those two sit in the same office of that building.
Theres no Oh, its your fault, Oh, I need this, Oh I cant get that because
Facilities Its one voice, one answer. And we involve IT in the design. Hell,
the servers move the air. Part of our design incorporates the IT side because thats
how we move the air. The fans are only to get it past the filters if we need it.
And during recirculation thats not even needed. So, this was a true partnership
between IT and Facilities from conception to commissioning.
Paul: I would say the same thing on the project side. Ive been involved in
construction projects beforenot in the capacity I was here, buta lot of times
your decisions are made by a design engineer who is not an operator, who hasnt
been an operator. Not that theyre wrong, but without an operators input to
challenge some of the assumptions if youre not involved in the process at that
point you may get handed a facility where theres a disconnect between the
operating principles and the design principles.
Bob, being the project manager, was really able to facilitate a lot of that. The
decisions that were made during the construction phase and getting the input
from the operations teams so that the decisions made were what the operations
team wanted to operate. Not what the mechanical design engineer or electrical
design engineer wanted to implement because that was their principles. More so
than any other project that Ive worked on it really integrated into the design phase
and so you knew what you were getting at the end, which is almost a luxury.
356
Nick: Heres an example. People stand around a table and say Lets go build
The Art of the Data Center
a Data Center and then they hire someone to do it. And what happens is,
that person that actually designs your Data Center is a guy that sits in a room
whacking on a keyboard on a BMS (building management system) that decides
how youre going to operate your building. Thats reality. Reality is that somebody
who has a history of building legacy systemsif he even built a Data Center
beforeis the one coding your BMS or coding your controls.
We took that away. We said You know what? We want control of that, and we
want to be able to say how its going to work. That was a fundamental design
change for us. We took total control. And we didnt allow somebody else to come
in and decide what they felt we needed to do. We just did it.
It was a change in the industry. It was very complicated at first. This guy was
saying What about my years of experience? Well I dont care. Because I dont
need to know how a fundamental Data Center works, I need to understand how
my Data Center works. And so thats how that kind of change came about.
Chris: I think an ongoing theme is the idea that the status quo isnt necessarily
going to work for the Data Center of the future. And yet another good quality
of the modular design is being able to continue to push the status quo. But, the
status quo is tied to a lot of operations systems and BMS to a lot of things. And
so challenging in design in some ways is just the first step.
Any final advice that you would offer someone who is taking on a
major Data Center project?
Nick: Yeah. Get the operators involved. Thats my best advice. What happens is
the design engineers design things the way they have done for years. So get the
operations team involved early.
The status quo isnt necessarily going to work for the Data Center
of the future.
357
Figure 18-6
18: Yahoo!
The U.S. Department of Energy awarded Yahoo! a
$9.9 million sustainability grant for its computing
coop design.
Figure 18-7
Outside air is drawn in, filtered, and then moved into the Data Center
with fans.
Figure 18-8
There are no raised floors in the data halls, saving material and
construction costs.
Figure 18-9
Yahoo! made sure to involve its Data
Center operators when developing its
computing coop.
Image by Christian Richters Photography.
Appendix
Although this book is aimed at a general audience, a handful of Data Center
industry terms crept into the conversations. Here is a brief overview of them, and
why theyre significant.
keep Data Center computing equipment operating for a short period of time
if primary (utility) power fails. Conventional UPS systems consist of batteries;
others employ a spinning flywheel.
VESDA (Very Early Smoke Detection Array)An air-sampling device that uses
a laser to check for smoke particles.