Applied Thermal Engineering: Jackson Braz Marcinichen, Jonathan Albert Olivier, John Richard Thome

Download as pdf or txt
Download as pdf or txt
You are on page 1of 16

Applied Thermal Engineering xxx (2012) 1e16

Contents lists available at SciVerse ScienceDirect

Applied Thermal Engineering


journal homepage: www.elsevier.com/locate/apthermeng

On-chip two-phase cooling of datacenters: Cooling system and energy


recovery evaluation
Jackson Braz Marcinichen*, Jonathan Albert Olivier, John Richard Thome
Laboratory of Heat and Mass Transfer (LTCM), Faculty of Engineering (STI), École Polytechnique Fédérale de Lausanne (EPFL), Station 9, CH-1015 Lausanne, Switzerland

a r t i c l e i n f o a b s t r a c t

Article history: Cooling of datacenters is estimated to have an annual electricity cost of 1.4 billion dollars in the United
Received 5 May 2011 States and 3.6 billion dollars worldwide. Currently, refrigerated air is the most widely used means of
Accepted 5 December 2011 cooling datacenter’s servers, which typically represents 40e45% of the total energy consumed in
Available online xxx
a datacenter. Based on the above issues, thermal designers of datacenters and server manufacturers now
seem to agree that there is an immediate need to improve the server cooling process. The goal of the
Keywords:
present study is to propose and simulate the performance of a novel hybrid two-phase cooling cycle with
Datacenter
micro-evaporator elements (multi-microchannel evaporators) for direct cooling of the chips and auxil-
Microprocessor
Hybrid two-phase cooling cycle
iary electronics on blade server boards (savings in energy consumption of over 60% are expected).
Micro-evaporator Different working fluids were considered, namely water, HFC134a and a new, more environmentally
Power plant friendly, refrigerant HFO1234ze. The results so far demonstrated that the pumping power consumption is
Energy recovery on the order of 5 times higher for the water-cooled cycle. Additionally, a case study considering the
hybrid cooling cycle applied on a datacenter and exploring the application of energy recovered in the
condenser on a feedwater heater of a coal power plant was also investigated (modern datacenters require
the dissipation of 5e15 MW of heat). Aspects such as minimization of energy consumption and CO2
footprint and maximization of energy recovery (exergetic efficiency) and power plant efficiency are
investigated.
Ó 2011 Elsevier Ltd. All rights reserved.

1. Introduction technologies is rejected as waste into the atmosphere. Therefore,


reusing this waste heat can potentially not only reduce the overall
Reduction of primary energy consumption is strongly required operating costs, but also the carbon footprint of the datacenter.
to mitigate global warming. To achieve this objective, the use of Most datacenters make use of air cooling technologies to ensure
waste heat thermal energy or renewable energy should be the correct running of the servers contained within. The limits of air
increased in energy conversion processes, reducing fossil fuel cooling, however, are being approached due to the performance
consumption. Under the current efficiency trends, the energy usage increase in microprocessors, which will have heat fluxes in the
of datacenters in the US is estimated to be more than 100 billion order of 100 W/cm2. This trend is shown in Fig. 1 with the red dots
kWh by 2011, which represents an annual cost of approximately obtained from [49]. It was shown that air has a maximum heat
$7.4 billion [15]. With the introduction of a proposed carbon tax in removal capacity of about 37 W/cm2 [47]. The problem is made
the US [23], the annual costs could become as high as $8.8 billion by worse with servers being more densely packed, such as the blade
2012, increasing annually. centers of IBM [20], which could see racks generating in excess of
Cooling of datacenters can represent up to 45% [22] of this total 60 kW of heat. Today’s datacenters are designed for cooling
consumption using current cooling technologies (air cooling). In capacities in the order of 10e15 kW per rack [48]. Hence, if data-
the US, this relates to an estimated 45 billion kWh usage by 2011 centers want to take advantage of increasing computer power
with an annual cost of $3.3 billion, or $4 billion with the inclusion of having a smaller surface footprint, other solutions to air cooling are
carbon tax. just for cooling. A problem with a datacenter is that all required.
the energy consumed is converted into heat, which with current One solution is to make use of on-chip cooling, where the
primary coolant, a liquid or refrigerant, is brought right up to the
main processor and auxiliary electronics and is used as a replace-
* Corresponding author. Tel.: þ41 21 693 5894; fax: þ41 21 693 5960. ment to air. Recent publications show the development of primarily
E-mail address: jackson.marcinichen@epfl.ch (J.B. Marcinichen). four competing technologies for the cooling of chips: microchannel

1359-4311/$ e see front matter Ó 2011 Elsevier Ltd. All rights reserved.
doi:10.1016/j.applthermaleng.2011.12.008

Please cite this article in press as: J.B. Marcinichen, et al., On-chip two-phase cooling of datacenters: Cooling system and energy recovery
evaluation, Applied Thermal Engineering (2012), doi:10.1016/j.applthermaleng.2011.12.008
2 J.B. Marcinichen et al. / Applied Thermal Engineering xxx (2012) 1e16

Nomenclature Acronyms
AE auxiliary electronics
CHF critical heat flux [Wcm2]
Roman COP coefficient of performance [-]
E_ d rate of exergy destruction due to irreversibilities CPU central processing unit
within the control volume, W EEV electric expansion valve
e_ fi, e_ fe inlet and outlet flow exergies, J/kg iHEx internal heat exchanger
m _ i; m
_e inlet and outlet mass flow rate, kg/s LA liquid accumulator
Q AE heat load associated with the auxiliary electronics, W LP liquid pump
Q cond heat transfer rate at the condenser, W LPR low pressure receiver
Q_ j heat transfer rate, W ME micro-evaporator
Q ME heat load associated with the microprocessor, W MPAE microchannel cold plate applied on auxiliary
Q sub heat transfer rate at the subcooler, W electronics
Tj instantaneous temperature,C PCV pressure control valve
T0 dead state temperature,C SP_W single-phase liquid water in the ME
W _ energy transfer rate by work, W TCV temperature control valve
CV
WLP_or_VC pumping power consumption of the driver, which can TP two-phase
be the liquid pump or the vapor compressor, W TP_HFC134a two-phase HFC134a in the ME
TP_HFO1234ze two-phase HFO1234ze in the ME
Subscripts VC vapor compressor
cond condenser VSC variable speed compressor
sub subcooler

single-phase flow, porous media flow, jet impingement cooling and heat also implies that chip temperatures are much more uniform. It
microchannel two-phase flow [1]. [25] showed that the use of such has been shown that heat fluxes as high as 300 W/cm2 [40] can be
new technology for cooling of chips could produce savings in achieved using a two-phase refrigerant while maintaining chip
energy consumption of over 60% [1] highlighted that the most temperatures below their critical value. Heat fluxes of 1000 W/cm2
promising of the four technologies was microchannel two-phase have been reached using refrigerants [24], although this was under
cooling. subcooled boiling with mass fluxes and pressure drops being an
Single-phase microchannel chip cooling has its apparent order of magnitude higher than microchannel two-phase boiling. It
advantage in that it is relatively easy to use. This technology is was further shown that heat fluxes of 180 W/cm2 can be removed
currently being used in the recently started Aquasar [16], which is with a saturation temperature as high as 60  C, while maintaining
an IBM blade center converted to make use of a water-cooled cycle. the chip temperature below 85  C [27]. A refrigerant has the
However, water has many disadvantages in that it requires a high advantage of being a dielectric fluid with a long successful history
pumping power to keep temperature gradients on the micropro- in industrial applications, is inert to most engineering materials,
cessor to within acceptable limits. Furthermore, water presents readily available and relatively inexpensive. New refrigerants also
a problem with its high freezing point, microbe formation, corro- have a negligible impact on the environment.
sive properties, electrical conductivity and erosive nature due to The main advantage of making use of on-chip cooling for the
high fluid velocities. cooling of datacenters is that the heat gained from cooling the chips
Two-phase microchannel flow takes advantage of the latent can be easily reused elsewhere. This is because the heat removal
heat of the fluid, which is much more effective in removing heat process is local to the chip and the fact that the heat carrying
than when using the sensible heat of a single-phase fluid. The latent capacity of the refrigerant is much higher than that of air, thus
minimizing any losses to the environment. The opportunity thus
exists to reuse the datacenter’s waste heat in secondary applica-
tions instead of being disposed of into the atmosphere. This not
only has the potential to reduce datacenter energy costs, but also to
reduce its carbon footprint and hence, its environmental impact.
A problem with recovering the heat of the datacenter is not in
the quantity of heat available, but rather in the quality of heat.
Currently, heat is being ejected into the atmosphere at tempera-
tures of about 40  C when using traditional air cooling methods.
This is because chip temperatures are being cooled at 15e20  C.
Due to the effective cooling of chips when using on-chip cooling,
fluid approach temperatures of about 60  C can be realized, while
removing high heat fluxes and keeping chip temperatures below
85  C [27]. Although the temperature, and hence the quality of the
heat is higher, the applications for using this heat are still limited.
Making use of a vapor compression cooling cycle, where
temperatures in excess of 90  C can be realized, can increase the
quality of the heat, i.e., the exergy, even further. Exergy of a system
is the maximum useful energy that brings the system into the
equilibrium with its surroundings. After the system and
Fig. 1. Heat flux trend of commercial processors. Red markers from [49]. surroundings reach equilibrium, the exergy is zero. The potential

Please cite this article in press as: J.B. Marcinichen, et al., On-chip two-phase cooling of datacenters: Cooling system and energy recovery
evaluation, Applied Thermal Engineering (2012), doi:10.1016/j.applthermaleng.2011.12.008
J.B. Marcinichen et al. / Applied Thermal Engineering xxx (2012) 1e16 3

for this kind of heat not only increases the number of applications small charge (a few milliliters). For a baseline operating condition,
that can use it, but also increases its value. Vapor compression when the evaporator and condenser temperatures and the heat
cooling cycles for computer chips have been demonstrated load were 50  C, 90  C and 50 W, the coefficient of performance
successfully [34] and [55], although it was focused on a small (COP) obtained was 2.25. The COP reached 3.70 when the
system where the recovery of heat was not of concern. evaporator and condenser temperatures increased and decreased
In this context, the objective of the present study is primarily to by 10  C from the baseline conditions and the heat load was
propose and analyze potential two-phase cooling cycles able to reduced to 44 W. The small-scale refrigeration system achieved
maintain the temperature of the chip below its upper operating 25e30% of the Carnot efficiency (ideal COP for a Carnot cycle),
limit (about 85  C) and to recover energy from the cycle’s values comparable with those obtained in today’s household
condenser for reuse, such as for heating a building, residence, refrigerators.
hospital, district heating, etc. To do this, an in-house integrated [55] designed, built and evaluated the performance of a miniature-
simulation code was developed which has the capability to design scale refrigeration system (MSRS) suitable for electronics cooling
the components and determine the performance of a hybrid cool- applications. Their MSRS had the following components: a commer-
ing cycle (liquid pump or vapor compressor as the driver of the cial small-scale compressor, a microchannel condenser, a manual
working fluid) under steady state conditions and for different needle valve as the expansion device, a cold plate microchannel
working fluids. The code is also able to evaluate the performance of evaporator, a heat spreader and two compressor cooling fans. A
the cooling cycles for single-phase and two-phase flow in the suction accumulator to avoid liquid flow to the compressor, an oil
micro-evaporators. filter to return oil to the compressor and guarantee good lubrication,
The new refrigerant HFO1234ze of Honeywell Inc. is considered and heat sources to simulate the chips were also installed. HFC134a
here as a potential substitute of HFC134a. This fluid has a “Global was the working fluid. System performance measurements were
Warning Potential” of only 6 against 1410 of HFC134a, i.e., it is conducted at evaporator temperatures from 10  C to 20  C and
considered as an immediate/future replacement for HFC134a. Both condenser temperatures from 40  C to 60  C. The cooling capacity of
HFC134a and HFO1234ze are dielectric fluids and thus compatible the system varied from 121 W to 268 W with a COP of 1.9e3.2 at
with electronics. HFC134a is currently the most widely used pressure ratios of 1.9e3.2. Their MSRS was able to dissipate CPU heat
refrigerant for refrigeration and air conditioning systems. fluxes of approximately 40e75 W/cm2 and keep the junction
Secondly, a case study is developed to demonstrate the overall temperature below 85  C for a chip size of 1.9 cm2. It was concluded
potential savings that can be made by implementing two-phase that a new compressor design for electronics cooling applications was
on-chip cooling within a datacenter. The investigation will take needed to achieve better performance of the system (the most
into account the benefits over the datacenter and a secondary significant losses occurred in the compressor, which was not designed
application of the waste heat. For this paper a coal fired thermal for the operating conditions of electronics cooling). It was also rec-
power plant will be analysed since more than 41% of the world’s ommended to study the development of an automatic expansion
energy is produced from these types of plants [57]. This plant was device and a suitable control strategy for the MSRS.
also chosen as it produces most of the greenhouse gases, with any [52] surveyed the advances in thermal modeling for flow boiling
potential improvements in efficiency translating to savings for both of low pressure refrigerants in multi-microchannel evaporators for
the power plant and the datacenter. cooling of microprocessors. According to them, multi-microchannel
evaporators hold promise to replace the actual air cooling systems
2. Literature review and can compete with water cooling to remove high heat fluxes,
higher than 300 W/cm2, while maintaining the chip safely below its
[18] have proposed a pumped liquid multiphase cooling system maximum working temperature, providing a nearly uniform chip
(PLMC) to cool microprocessors and microcontrollers of high-end base temperature [2] and minimizing energy consumption.
devices such as computers, telecommunications switches, high- Variables such as critical heat fluxes, flow boiling heat transfer
energy laser arrays and high-power radars. According to them, coefficients and two-phase friction factors were evaluated and
their system could handle applications with 100 W heat loads characterized as important design parameters to the micro-
(single computer chip) as well as applications with short time evaporator for high heat flux applications.
periods of kW heat loads (radar). Their PLMC consists of a liquid [53] simulated two-phase cooling elements for microprocessors
pump, a high performance cold plate (evaporator) and a condenser with micro-evaporation. Heat fluxes of 50 W/cm2 and 150 W/cm2
with a low acoustic noise air mover to dissipate the heat in the in a micro-evaporator with channels 75 mm wide, 680 mm high and
ambient air. A comparison between a single-phase liquid loop 6 mm long with 100 mm thick fins were simulated for flow boiling.
(water) and the system proposed with HFC134a was made for The size of the chip was assumed to be 12 mm by 18 mm and the
a 200 W heat load. The HFC134a system had a mass flow rate, micro-evaporator was considered with the fluid inlet at the
a pumping power and a condenser size that were 4.6, 10 and 2 centerline of the chip and outlets at both sides, i.e., a split flow
times smaller than the water-cooled system. The coolant temper- design to reduce the pressure drop and increase the critical heat
ature rise was 10  C for the water but negligible for HFC134a. They flux. Results of pumping power, critical heat flux, and junction and
emphasized the significant benefits from efficiency, size and weight fluid temperatures were generated for HFC134a at an inlet satura-
that were provided with the PLMC solution. tion temperature of 55  C (chosen to allow for heat recovery). The
[34] designed and built a small-scale refrigeration system following conclusions were reached: i) the influence of mass flux on
applicable for a notebook computer. The system included a mini- the fluid, chip and wall temperatures was small, ii) for the heat flux
compressor, a microchannel condenser, a microchannel evaporator of 150 W/cm2, the chip temperature was 70  C or less, i.e., well
and a capillary tube as the throttling device and is considered to be below its operational limit of 85  C, iii) the junction-to-fluid
the first refrigeration system developed that can fit within the tight temperature difference was only 15 K for the heat flux of
confines of a notebook and operate with high refrigeration effi- 150 W/cm2, which is lower than that with liquid cooling systems,
ciencies. HC600a (isobutane) was the working fluid, chosen from an iv) the fluid working temperature could still be raised by 10 K to
evaluation of 40 candidate refrigerants. According to them, HC600a a junction temperature of 80  C while rejecting heat at 65  C for
presented the best efficiency at a low pressure ratio and was readily reuse, and v) the critical heat flux increased with the mass flux and
available. Although it is flammable the system only required a very the lower limit was about 150 W/cm2 for 250 kg/m2s. The channel

Please cite this article in press as: J.B. Marcinichen, et al., On-chip two-phase cooling of datacenters: Cooling system and energy recovery
evaluation, Applied Thermal Engineering (2012), doi:10.1016/j.applthermaleng.2011.12.008
4 J.B. Marcinichen et al. / Applied Thermal Engineering xxx (2012) 1e16

width had a significant effect on the wall and junction tempera- heat fluxes are achieved with a smaller EEV opening and higher
tures, and there was a turning point at about 100 mm when heat input supplied to the accumulator and iii) a trade-off between
considering 1000 kg/m2s of mass flux and 150 W/cm2 of base heat the system COP and CHF is necessary to prevent the device burnout,
flux, at which these temperatures reached a minimum. For the i.e., imposed heat flux must be lower than the CHF considering
same mass flux and base heat flux, the reduction of channel width a safety margin. Finally, they presented a preliminary validation of
also reduced the energy consumption to drive the flow (pumping the model with initial experimental data showing a satisfactory
power). prediction ability of the model. The authors do not mention
From a system viewpoint [53], showed an approximate anything about the geometry assumed for the evaporators.
comparison of performances of liquid water cooling versus two- The present study is a continuation of the earlier study devel-
phase cooling. For the same pumping power consumption to oped by [28]. They proposed a two-phase cooling cycle considering
drive the fluids, two-phase cooling allowed the chip to operate a liquid pump as the driver of the fluids, a micro-evaporator for
about 13 K cooler than water cooling or it could operate at the same cooling the chip and its auxiliary electronics, and a microfin tube-
junction temperature but consume less pumping power using in-tube condenser for heat recovery. A standard length (30 cm)
a lower refrigerant flow rate. The two-phase cooling system and internal diameter (3 mm) was defined for the pipes joining the
appeared to be more energy-efficient than classical air cooling or components, which were assumed to be only straight and hori-
direct liquid cooling systems while also exhausting the heat at zontal, i.e., the effects of bends and static height difference between
higher reusable temperatures. Regarding the choice between components and pipes were ignored. The performance of the
a pump and a compressor as the driver for a micro-evaporation cooling cycle considered 3 different working fluids; HFC134a,
heat sink system, they emphasized that the choice depends on HFO1234ze and water (in an analogous single-phase cooling cycle).
the economic value of the reused energy. The system with The results showed that for a design of the cooling cycle so that the
a compressor is ideal for energy reuse because of the higher heat total pressure drop is about 1 bar, the liquid water cooling cycle had
rejection temperature; however the additional energy consumed a pumping power consumption 16.5 times that obtained for the
by the compressor compared to the pump has to be justified by the two-phase HFC134a cooling cycle. When comparing with the
reuse application. HFO1234ze cooling cycle, which showed a total pressure drop of
[31] evaluated the performance of a multi-microchannel copper 1.2 bar, the difference drops to 13.2 times. These results can be
heat sink with respect to critical heat flux (CHF) and two-phase considered a differential when compared with demonstration
pressure drop. A heat sink with 29 parallel channels (199 mm projects, such as that for the new supercomputer called AQUASAR
wide and 756 mm deep) was tested experimentally with a split flow [16], which considers the implementation of a liquid water cooling
system with one central inlet at the middle of the channels and two cycle on a rack cabinet with power consumption of around 10 kW. It
outlets at either end. Three working fluids were tested (HFC134a, is important to mention that the simulations presented were
HFC236fa and HFC245fa) and also the parametric effects of mass considered as a benchmark and that the energetic comparison
velocity, saturation temperature and inlet temperature. The anal- should be applied to an actual server’s specifications, which is the
ysis of their results showed that a significantly higher CHF was subject that is considered here. Additionally, a case study consid-
obtainable with the split flow system compared to the single inlet- ering a vapor compression and a liquid pumping cooling systems
single outlet system [41], providing also a much lower pressure applied on a datacenter and exploring the application of energy
drop. For the same mass velocity, the increase in CHF exceeded 80% recovered in the condenser on a feedwater heater of a coal power
for all working fluids evaluated due to the shorter heated length of plant will be presented. Aspects such as the minimization of energy
a split system design. For the same total refrigerant mass flow rate, consumption and CO2 footprint and the maximization of energy
an increase of 24% for HFC134a and 43% for HFC236fa were ob- recovery and power plant efficiency are investigated.
tained (no comparable data were available for HFC245fa). They
concluded that the split flow system had the benefit of much larger 3. Hybrid two-phase cooling cycle
CHF values with reduced pressure drops and further developments
in the design of split flow system could yield an interesting ener- Fig. 2 depicts the proposed hybrid two-phase cooling cycle, i.e.,
getic solution for cooling of computer chips. a multi-purpose cooling cycle able to interchange the cycle driven
[58] developed a steady state model of a refrigeration system for by a liquid pump or a vapor compressor. The change of cycle would
high heat flux electronics cooling. The refrigeration system be accomplished by means of shut-off valves 1 to 7 (SOV). The
proposed consists of multiple evaporators (microchannel tech- choice of the cooling cycle would depend on the demand for heat
nology), liquid accumulator with an integrated heater, variable recovery, or whether cycle maintenance is required (repair of the
speed compressor, condenser and electric expansion valves (EEV). compressor or pump with one mode as a backup to the other
To obtain more efficient heat transfer and higher critical heat flux, mode). The microprocessors cannot operate without cooling and
the evaporators were considered to operate only with two-phase thus the interchangeability of the cycles represents a safety
flow. To guarantee the safe operation of the refrigeration system mechanism in case of failure of the pump or compressor. The cons
the authors considered the presence of an integrated heater- of the hybrid cycle would mainly be the higher initial cost but
accumulator to fully evaporate the two-phase flow coming out of certainly the advantages gained (system online reliability,
the evaporator, which naturally represents a decrease of the cycle controllability, cycle interchangeability and flexibility in heat
COP. A parametric study to evaluate the effects of external inputs on recovery) may prove to justify this. Furthermore, this hybrid cycle
the system performance (secondary fluid temperature in the represents a plug-and-play option where any one of the three
condenser, evaporator heat load, compressor speed, EEV cycles can be installed based on the particular application to
percentage opening and heat supplied to the accumulator) and standardize the design, thus minimizing engineering costs.
a Pareto optimization to find the optimal system operating condi- The goal is to control the chip temperature to a pre-established
tions were also developed. A heat load of 1500 W and 2500 W, level by controlling the inlet conditions of the micro-evaporator
which represent respectively a heat flux of 94 kW/m2 and (pressure, subcooling and mass flow rate). It is also imperative to
156.6 kW/m2 were considered. The main points observed were: keep the micro-evaporator (ME) outlet vapor quality below that of
i) the system COP can be improved without compromising the the critical vapor quality, which is associated with the critical heat
critical heat flux when handling higher heat flux, ii) higher critical flux. The critical vapor quality is determined from the critical heat

Please cite this article in press as: J.B. Marcinichen, et al., On-chip two-phase cooling of datacenters: Cooling system and energy recovery
evaluation, Applied Thermal Engineering (2012), doi:10.1016/j.applthermaleng.2011.12.008
J.B. Marcinichen et al. / Applied Thermal Engineering xxx (2012) 1e16 5

Fig. 2. Hybrid two-phase cooling cycle.

flux [39] and an energy balance on the ME. Due to this limitation, avoids potential problems of mal-distribution in each ME line
additional latent heat is available for further evaporation, which due to non-uniform heat load on the microprocessors and bad
can be used by other low heat flux generating components in the pipeline designs. Preliminary experimental evaluation can be
blade server. found in [30]; where one SMV controlling the mixture (junc-
Another parameter that must be controlled is the condensing tion of piping) outlet vapor quality of two MEs proved to be
pressure (condensing temperature). The aim, during the winter, is feasible and efficient.
to recover the energy dissipated by the refrigerant in the condenser c) Micro-evaporator (ME): transfers the heat generated by the
to heat buildings, residences, district heating, etc. In order to microprocessor to the refrigerant.
accomplish this, the idea is to use a variable speed compressor d) Microchannel cold plate for auxiliary electronics (MPAE):
(VSC) and an electric expansion valve (EEV), as will be discussed additional component used to cool the auxiliary electronics
below. using the remaining latent heat, which is available due to the
Fig. 3 depicts the two-phase cooling cycle where the flow rate is limitations enforced on the micro-evaporator.
controlled by a liquid pump. The components considered and their e) Pressure control valve (PCV): controls the condensing pressure.
main functions are presented below: f) Condenser: counter-flow tube-in-tube exchanger.
g) Liquid accumulator (LA): guarantees that there is only satu-
a) Variable speed liquid pump: controls the mass flow rate rated liquid at the subcooler inlet, independent of changes in
circulating in the system. thermal load.
b) Stepper motor valve: controls the liquid flow rate to control the h) Temperature control valve (TCV): controls the subcooling at the
outlet vapor quality in each micro-evaporator (0%e100%). It inlet of liquid pump.

This cycle is characterized in having a low vapor quality at the


ME outlet, a low condensing temperature and a high overall

Fig. 3. Schematic of the liquid pump cooling cycle. Fig. 4. Schematic of the vapor compression cooling cycle.

Please cite this article in press as: J.B. Marcinichen, et al., On-chip two-phase cooling of datacenters: Cooling system and energy recovery
evaluation, Applied Thermal Engineering (2012), doi:10.1016/j.applthermaleng.2011.12.008
6 J.B. Marcinichen et al. / Applied Thermal Engineering xxx (2012) 1e16

efficiency. This is a good operating option when the energy dissi- load on the blade, but will have a larger surface area compared to
pated in the condenser is not recovered, typically during the the CPU and thus a lower heat flux.
summer season. However, the heat can still be recovered if there is Finally, when considering an entire rack, a very sizable heat load
an appropriate demand for low quality heat (low exergy). is generated, which represents a good opportunity to recover the
Fig. 4 shows a two-phase cooling cycle where a vapor heat rejected. In this case, reuse of the heat removed from the
compressor is the driver of the working fluid. The components blades for a secondary application will greatly reduce the CO2
considered and their main functions are: footprint of the system. For example, if we consider a datacenter
with 50 vertical racks, where each rack has 64 blades and each
a) Variable speed compressor: controls the ME inlet pressure and blade dissipates 300 W, the total potential heat to be recovered will
consequently the level of inlet subcooling. be 0.96 MW. Such a heat recovery system requires a secondary heat
b) Pressure control valve (PCV): controls the condensing pressure. transfer fluid to pass through all the condensers (either water or
c) Condenser: counter-flow tube-in-tube exchanger. a refrigerant) and then transport the heat to its destination. Addi-
d) Liquid accumulator: guarantees that there is only saturated tional information on such cycles can be found in the previous work
liquid at the internal heat exchanger (iHEx1) inlet. of the [29].
e) Internal heat exchanger liquid line/suction line (iHEx1):
increases the performance of the cooling system. 4. Simulation code
f) Electric expansion valve (EEV): controls the low pressure
receiver level. An in-house two-phase cooling cycle simulation code was
g) Low pressure receiver (LPR): this component can be seen as developed to design and evaluate the performance of the liquid
a second internal heat exchanger liquid line/suction line, which pump and vapor compressor cooling cycles under steady state
increases the EEV inlet subcooling and allows an overfeed to conditions. The simulation code is able to design the condenser and
the ME since the ME outlet returns to this receiver. subcooler, to evaluate the performance of the ME and various
h) Stepper motor valve: controls the liquid flow rate to control the component coolers for a given heat load, and to calculate the
outlet vapor quality in each micro-evaporator (0%e100%). pumping power consumption to drive the cooling cycle. The pres-
i) Micro-evaporator (ME): transfers the heat away from the sure drop of each component and the piping are also calculated.
microprocessor. Table 1 shows the principal methods implemented in the code.
j) Microchannel cold plate for auxiliary electronics (MPAE): cools In summary the following assumptions are considered:
the auxiliary electronics.
 steady state simulation;
This cycle is characterized by a high condensing temperature  adiabatic piping;
(high heat recovery potential), a high range of controllability of the  isentropic pumping and compression;
ME inlet subcooling (characteristic of systems with VSC and EEV),  isenthalpic expansion (adiabatic);
and potentially a medium overall efficiency when compared with  uniform heat load on the microprocessors (no hotspots);
the liquid pumping cooling cycle. This is a good operating option  uniform flow distribution on the MEs;
when the energy dissipated in the condenser is recovered for other  one-dimensional heat transfer in the ME
use, typically during the winter season when considering a district
heating application or the entire year for a feedwater heater of The input data required to run the simulation code are: (i) the
a coal power plant (high exergy). geometrical parameters of ME’s and heat exchangers, (ii) the heat
It is worth mentioning that the applicability of these cooling load on the ME’s and MPAE’s, (iii) the evaporating temperature and
cycles is not restricted to only one microprocessor but can be subcooling at the ME inlet, the condensing temperature at the
applied to blade servers and clusters, which may have up to 64 condenser inlet (only for the vapor compression cooling cycle), the
blades per rack cabinet. Each blade, such as that shown in Fig. 5, can water temperature at the condenser and subcooler inlet and outlet
have two (or more) microprocessors with a heat generation (secondary fluid flowing in the annulus), and (iv) the length and
capacity higher than 150 W. If the auxiliary electronics (memories, direction of the pipes and elbows joining the components.
DC/DC, etc.) on the blade are included, the total heat generation per The analysis took into account the thermal performance, the
blade can be higher than 300 W. Thus, the microchannel cold plate pumping power consumption and the total pressure drop of the
(MPAE) described in the cooling cycles has the function to cool the cooling cycle. A comparison of 5 simulated cases considering
auxiliary electronics that can represent about 60% of the total heat different working fluids and cooling cycles was made. For simulated
cases 1 and 4, two-phase (TP) HFC134a and single-phase liquid
water (SP_W) in the ME, the design considered for the liquid pump
cooling system was such that the total pressure drop was about
1.5 bar. The design constraint imposed on the condenser and the
subcooler was that the pressure drops in the working and
secondary fluids were, respectively, 0.05 bar and 1 bar. Cases 2, 3
and 5 considered the same heat exchanger geometries
(ME, condenser and subcooler) as defined for case 1. The difference
in cases 2, 3 and 5 with respect to case 1 are that for case 2 the
internal diameter of piping on the blade was reduced from 3 mm to
2 mm, for case 3 the working fluid was changed from HFC134a to
HFO1234ze and for case 5 the vapor compression cooling cycle was
considered.
A blade server with 16 blades was taken into consideration for
the total heat load. Each blade, for example as that showed in Fig. 5,
Fig. 5. Typical blade with two microprocessors and a heat generation capacity higher presents two electronic systems in parallel and with each system
than 300 W. being composed of one microprocessor (60 W of heat load) and the

Please cite this article in press as: J.B. Marcinichen, et al., On-chip two-phase cooling of datacenters: Cooling system and energy recovery
evaluation, Applied Thermal Engineering (2012), doi:10.1016/j.applthermaleng.2011.12.008
J.B. Marcinichen et al. / Applied Thermal Engineering xxx (2012) 1e16 7

Table 1 Table 2
Methods in the code. Input data.

Component Type Method Component Working fluid Input data


ME Multi-Microchannel Heat transfer coefficient Auxiliary All of them 55.6 W per half blade
by [54]. Electronics
Pressure drop as ME HFC134a and Inlet evaporating temperature ¼ 60  C
suggested by [45]. HFO1234ze Inlet Subcooling ¼ 0  C
Critical heat flux by [44]. Outlet vapor quality ¼ 30%
Condenser inner tube: Spiral m-fin Heat transfer coefficient QME ¼ 60 W per ME
(tube-in-tube) (single-phase flow) by [32]. Water Inlet temperature ¼ 60  C
Pressure drop by [32]. Outlet temperature ¼ 62  C
inner tube: Spiral m-fin Heat transfer coefficient QME ¼ 60 W per ME
(two-phase flow) by [9]. Condenser HFC134a and Inlet condensing temperature ¼ 95  C
Pressure drop by [9]. (tube-in-tube) HFO1234ze (case 5)
annulus: Smooth Heat transfer coefficient Outlet vapor quality ¼ 0%
(single-phase flow) by [13]. Secondary fluid: Inlet temperature ¼ 15  C
Pressure drop by [7]. water Outlet temperature ¼ inlet condensing
Subcooler inner tube: Ribbed Heat transfer coefficient temperature 10  C
(tube-in-tube) (single-phase flow) by [43]. Outlet temperature ¼ inlet condensing
Pressure drop by [43]. temperature 5  C (case 5)
annulus: Smooth Heat transfer coefficient Subcooler HFC134a and Inlet vapor quality ¼ 0%
(single-phase flow) by [13]. (tube-in-tube) HFO1234ze
Pressure drop by [7]. Secondary fluid: Inlet temperature ¼ 15  C
Straight horizontal adiabatic (single-phase flow) Pressure drop by [7]. Water Outlet temperature ¼ subcooler inlet
pipes adiabatic (two-phase flow) Pressure drop by [36]. temperature 10  C
Straight vertical adiabatic (single-phase flow) Pressure drop by [7] Compressor HFC134a Isentropic compression
pipes (upward) and [3]. Condensing temperature ¼ 95  C
adiabatic (two-phase flow) Pressure drop by [6,51]; Liquid Pump All of them Isentropic pumping
[5] and [26]
Straight vertical adiabatic (single-phase flow) Pressure drop by [7]
pipes and [3].
(downward) adiabatic (two-phase flow) Pressure drop by [5,6] determine the total energy balance of the cycle come from the
and [42]. linkage to the methods shown earlier in Table 1.
Elbow (horizontal) adiabatic (single-phase flow) Pressure drop by [50]. Eq. (1) shows the total energy balance of the cooling cycle:
adiabatic (two-phase flow) Pressure drop by [4].
Elbow (vertical) adiabatic (single-phase flow) Pressure drop by [50]
Q ME þ Q AE þWLP or VC ¼ Q cond þ Q sub (1)
and [3]. |fflfflfflfflfflfflfflffl{zfflfflfflfflfflfflfflffl}
adiabatic (two-phase flow) Pressure drop by [3]. Heat load

QME and QAE are the heat loads associated with the microprocessor
auxiliary electronics (55.6 W of heat load). Fig. 6 shows the rack and the auxiliary electronics, which are transferred respectively by
cabinet designed, with the piping configurations and components the ME and MPAE. WLP_or_VC is the pumping power consumption of
for the liquid pump cooling cycles with water and volatile working the driver, which can be the liquid pump or the vapor compressor.
fluid (a, b) and (c) the vapor compression cooling cycle. Finally, Qcond and Qsub are the heat transfer rate at the condenser
In summary the following comparisons were made: i) SP_W and subcooler.
versus TP_HFC134a, ii) TP_HFO1234ze versus TP_HFC134a, both It is worth pointing out that the condenser is a tube-in-tube heat
comparisons for the liquid pump cooling cycle and iii) liquid pump exchanger with spiral microfins on the internal surface of the inner
cooling cycle versus vapor compression cooling cycle, both cycles tube with a smooth external surface. For the tube-in-tube sub-
with TP_HFC134a. cooler, the internal surface of the inner tube is again microfinned
Table 2 shows the input data considered for each working fluid and smooth on the external surface. For both heat exchangers the
evaluated. The other thermodynamic parameters required to objective was to find its length, while the other geometrical

Fig. 6. Rack cabinet e Frontal view.

Please cite this article in press as: J.B. Marcinichen, et al., On-chip two-phase cooling of datacenters: Cooling system and energy recovery
evaluation, Applied Thermal Engineering (2012), doi:10.1016/j.applthermaleng.2011.12.008
8 J.B. Marcinichen et al. / Applied Thermal Engineering xxx (2012) 1e16

parameters were considered to be fixed. Water was considered as Table 4


the secondary fluid in the annulus. Geometrical parameters of the ME.

Tables 3 and 4 show the geometrical parameters considered for Micro-evaporator


the condenser, subcooler and ME. For the ME the same geometrical Fin height [mm] 1700 Length [mm] 13.5
parameters were considered in all cases with the only exception Fin width [mm] 170 Width [mm] 18.5
being for case 4 (SP_W cooling cycle) where the orifice distribution Channel width [mm] 170 Area [cm2] 2.5
Base thickness [mm] 1 Material Copper
plate, normally used at the inlet of ME’s to avoid problems of mal-
Split flow 1 inlet and 2 outlets
distribution in the channels in two-phase flow [1], was not
considered.
It is important to mention that for case 4 a value of 2 K was pressure drop, heat transfer rate in the heat exchangers and their
simulated as the maximum axial rise in the chip’s temperature from calculated length, internal diameter of piping, water mass flow rate,
inlet to outlet of the ME, which also represents the rise in the water outlet temperature at the heat exchangers and pressure drop
temperature from inlet to outlet. The actual temperature rise could (secondary fluid) are shown. It is important to remember that the
be more, depending on the computer manufacturer’s design spec- SP_W cooling cycle does not consider the condenser and liquid
ifications. Increasing this temperature difference will decrease the accumulator, which are not required in that cycle (see Fig. 6).
water flow rate for its simulation, and hence also reduce its pressure Comparing cases 1 and 4, TP_HFC134a and SP_W liquid pump
drop and pumping power accordingly, but will increase the local cooling cycles, it is worth noting the difference regarding the
temperature of the microprocessor at the exit. On the other hand, internal diameter of piping and pumping power consumption. Case
unless the water is charged into the server’s cooling system on site, 4 presents the largest values, 7.7 mm and 35 W, respectively. A
then glycol must be added to the water circuit before shipment to larger pipe diameter was necessary to guarantee the 1.5 bar pres-
prevent freezing, which will increase the pressure drop by about sure drop defined as a design constraint, with the large pumping
50%. For the other TP cooling cycle cases, the maximum axial rise in power being a consequence of the high mass flow rate of water (see
the chip’s temperature obtained by the ME simulations was about Table 5 and Fig. 7). The pressure drop in the piping on the blade was
0.2 K, which is due to the near ideal matching of the tandem fall in 6 times lower for case 1. The total length of the heat exchangers,
the local flow boiling heat transfer coefficient and saturation shown in Table 7, was highest for case 1. However, since case 4
temperature (pressure) along the micro-evaporator. imposed greater inner and outer tube diameters due to the 0.05 bar
Table 5 shows the results obtained by the methods developed to and 1 bar pressure drop design constraints in the working and
evaluate the performance of the ME’s. The three-zone model [54] secondary fluid sides respectively, the volume occupied by the heat
was used for predicting the two-phase heat transfer since it was exchangers is highest for case 4 (24% higher than for case 1), which
shown to predict many fluids and geometries with good accuracy represents a higher material cost. Finally, it can also be observed for
[14], the numerically based model of [44] was used for critical heat all cases that the pressure drop in the piping represents more than
flux (CHF) calculations and the homogeneous model, in conjunc- 90% of the total pressure drop (viz. Fig. 8).
tion with the two-phase mixture viscosity model of [10]; was used Case 2 considers a smaller internal diameter of the piping on the
for two-phase pressure drops since it was found to predict micro- blade, i.e., 2 mm, which can be advantageous for installing the
channel pressure drops with relatively good accuracy [45]. cooling system on the blade due to the limited space available. The
The results show that a much higher mass flow rate of water is smaller diameter piping is also inherently more flexible making it
required for the SP_W cooling cycle than for the TP cooling cycles, easier to conform to the layout of the electronic components on the
which is justified by the latent heat of the refrigerants being 30e60 motherboard, which leads to less stress exerted on the blade. The
times the liquid specific heat of the water and because of the low comparison of the results between cases 1 and 2 shows an increase
maximum junction temperature rise defined as an input parameter. in pumping power and total pressure drop of about 9.4% and 7.1%,
The pressure drop is low for all the fluids, in part as a consequence respectively, although these values are still much lower than that
of the split flow design. For the outlet vapor quality considered obtained for the SP_W (case 4) presented earlier. Case 3, which
(30%), the predicted CHF was higher than 6 times the actual considers the new environmentally friendly working fluid
maximum heat flux of 24 W/cm2. This safety factor is more than HFO1234ze, with the heat exchangers also having the same
sufficient since the accuracy in predicting CHF is about 20%. geometries as in case 1, showed a small increase in pumping power
Tables 6 and 7 and Figs. 7 to 9 show the simulation results for consumption and total pressure drop when compared with case 1,
the 5 cases mentioned beforehand. The pressure drop in each being on the order of 23.4% and 16.2%, respectively.
component and piping, pumping power consumption, total Case 5 shows the simulation results of the TP_HFC134a vapor
compression cooling cycle, which considers the same piping
diameters and heat exchanger geometries defined for case 1. A
Table 3
significant increase in pumping power is observed compared to
Geometrical parameters/condenser and subcooler.
case 1. However, what is of interest is the lower secondary fluid
Cooling cycle Liquid pump Vapor compression
Working fluid HFC134a or HFO1234ze Water HFC134a Table 5
Heat exchanger Condenser Subcooler Subcooler Condenser ME performance.
Inner tube Spiral m-fin Ribbed Ribbed Spiral m-fin
Cooling cycle TP SP_W
Fin tip diameter 10.8 5.53 14.00 10.8
[mm] Working Fluid HFC134a HFO1234ze Water
Fin height [mm] 0.25 0.14 0.305 0.25 Inlet evaporating temperature 
[C] 60 60 e
Thickness of the 0.35 0.27 0.635 0.35 Outlet evaporating temperature [C] 60 60 e
tube [mm] Mass flow rate per ME [kg/h] 5.18 5.32 25.81
Helix angle [o] 18 16 27 18 Outlet vapor quality [%] 30 30 e
Apex angle [o] 50 e e 50 Inlet temperature [C] 60 60 60
Number of fins [-] 70 55 60 70 Outlet temperature  [C] e e 62
Outer tube Smooth Smooth Smooth Smooth Pressure drop [bar] 0.001 0.001 0.005
Internal diameter 17.52 8.72 21.33 17.52 Heat flux [W/cm2] 24 24 24
[mm] CHF [W/cm2] 154.4 165.6 e

Please cite this article in press as: J.B. Marcinichen, et al., On-chip two-phase cooling of datacenters: Cooling system and energy recovery
evaluation, Applied Thermal Engineering (2012), doi:10.1016/j.applthermaleng.2011.12.008
J.B. Marcinichen et al. / Applied Thermal Engineering xxx (2012) 1e16 9

Table 6
Cooling cycle piping designs and pressure drops on the blade and overall system.

CASES 1 2 3 4 5
Cooling Cycle Liquid pump Vapor compression
Working Fluid HFC134a HFC134a HFO1234ze Water HFC134a
Internal diameter of piping on the blade [mm] 3.0 2.0 3.0 3.0 3.0
Internal diameter of piping for single-phase flow [mm] 4.1 4.1 4.1 7.7 4.1
Internal diameter of piping for two-phase flow [mm] 7.0 7.0 7.0 e 7.0
Pressure drop in the piping on the blade [bar] 0.021 0.134 0.028 0.127 0.021
Total pressure drop [bar] 1.54 1.65 1.79 1.50 1.87

mass flow rate and heat exchanger volume and the higher outlet and typical efficiency values (20% for the pump and 60% for the
temperature of secondary fluid (viz. Table 7 and Fig. 9), when compressor) a power consumption of about 32 W and 1250 W
compared with all the cases evaluated. This implies that a lower are obtained for the pump and compressor, respectively.
pumping power for the secondary fluid is required while material Therefore, for the expected efficiency values, the power
costs of the condenser are lower due to the lower volume and that consumption of the compressor is 39 times higher than the
there exists a higher potential for energy recovery (high exergy). As liquid pump, compared to 116 times higher if 100% efficiency
mentioned beforehand, this cycle would be a good operating option were assumed (Fig. 7).
when the energy dissipated in the condenser is recovered for other
uses, for example into the feedwater heaters of thermal power
plants. It is also worth mentioning that the higher heat transfer 5. Datacenter and coal fired power plant synergy: case study
duty in the condenser (viz. Table 7) is associated with the additional
work imparted by the compressor on the fluid. This subject will be The case study is performed where it is considered that the
explored in the following section. datacenter uses on-chip cooling to cool the servers, with the heat
Finally, Fig. 10 shows the effect the driver’s isentropic efficiency captured being redistributed to a power plant. The datacenter will
has on the driver’s power consumption for cases 1 and 5. The be modeled as a cooling cycle consisting of an evaporator (on-chip
simulations assumed that the extra power associated with the cooling elements), a condenser, a fluid driver (compressor or liquid
inefficiency of the drivers was lost to the environment, i.e., it does pump), and an expansion valve in the case of a vapor compression
not have an effect on the cooling cycle’s thermodynamic conditions system. The power utility will be a thermal Rankine cycle consisting
and design declared beforehand. Due to the low viscosity of of a boiler, a high and low pressure turbine, a condenser, a low
HFC134a, 0.12 mm2/s at 60  C, the current pumps are not expected pressure and high pressure feedwater pump and a feedwater
to have high isentropic efficiency (slip and leakage reasons/lower heater. The feedwater heater receives heat from steam tapped after
than 20%), on the other hand conventional vapor compressors are the high pressure turbine. The optimal pressure for tapping the
expected to have 50%e70% isentropic efficiency [19] and [17]. It is steam will be calculated to obtain maximum thermal efficiency. The
also worth mentioning that such efficiency is highly dependent on datacenter waste heat will be injected into the Rankine cycle after
the manufacturer of the driver, economic aspects and thermody- the condenser and prior to the feedwater heater. This is shown
namic conditions, reasons which will not be explored in this paper. schematically in Fig. 11.
The results show that the driver’s isentropic efficiency has The operating conditions for both cycles are listed in Table 8,
a considerable impact on its power consumption. For expected where the values of a typical coal fired steam cycle power plant
reported by [56] are considered. A first-law analysis is performed
on the two cycles, showing their overall performances. The
Table 7 following assumptions are made:
Heat exchangers simulations.

CASES 1 2 3 4 5  no pressure drop in components;


Condenser  isentropic compression;
Length [m] 2.66 2.66 2.78 e 2.62  isentropic pumping;
Mass flow rate of 81.88 82.88 87.70 e 50.72  isenthalpic expansion;
secondary fluid [kg/h]  100% exchanger efficiency.
Pressure drop secondary 0.954 0.976 1.129 e 0.374
fluid side [bar]
Pressure drop working 0.044 0.045 0.060 e 0.021
fluid side [bar] 1000
e
746.5

Outlet temperature of 48.0 47.7 46.8 90.0


Power consumption [W]

secondary fluid  [C]


Heat transfer rate [W] 3176.65 3199.23 3274.01 e 4445.70
100
Subcooler
Length [m] 1.07 1.05 1.00 2.45 e
Mass flow rate of 13.60 13.14 11.51 84.34 e
35.0

secondary fluid [kg/h]


Pressure drop secondary 0.981 0.906 0.687 1.045 e 10
7.9

fluid side [bar]


7.0
6.4

Pressure drop working 0.054 0.053 0.055 0.053 e


fluid side [bar]
Outlet temperature of 47.9 47.6 46.5 53.2 e 1
secondary fluid  [C] HFC134a HFC134a HFO1234ze Water HFC134a
Heat transfer rate [W] 528.95 506.97 433.09 3734.20 e Case1 Case2 Case3 Case4 Case5
Total heat exchanger 705.2 704.0 729.9 875.5 631.6 LP LP LP LP VC
volume [cm3]
Fig. 7. Power consumption of the drivers (LP e Liquid pump, VC e Vapor compressor).

Please cite this article in press as: J.B. Marcinichen, et al., On-chip two-phase cooling of datacenters: Cooling system and energy recovery
evaluation, Applied Thermal Engineering (2012), doi:10.1016/j.applthermaleng.2011.12.008
10 J.B. Marcinichen et al. / Applied Thermal Engineering xxx (2012) 1e16

1600 80
1400 Case 5 70

Power consumption [W]

Power consumption [W]


1200 Case 1 60
1000 50

(Case 5)

(Case 1)
800 40
600 30
400 20
200 10
0 0
0 20 40 60 80 100 120
Isentropic efficiency [%]

Fig. 10. Driver’s power consumption vs. isentropic efficiency.

Fig. 8. Percentage breakdown of pressure drop by component.

5.1. Power utility

Heat captured in the datacenter can be reused by a power plant.


Since the waste heat of the datacenter is of a low quality, it can only
be inserted after the condenser of the power plant. This would then Fig. 11. Datacenter integrated in a power utility.
increase the temperature of the water leaving the condenser (46  C
typically) to a maximum temperature as defined by the condensing
temperature of the datacenter cycle. Therefore, any additional heat the power plant. By using a liquid pumping cycle in the datacenter,
added to the power plant’s cycle will result in less fuel needing to condensing temperatures of 60  C can be reached since this would
be burnt, thus saving fuel and reducing the CO2 footprint of the imply that the evaporating temperature on the chip is also 60  C.
power plant. For higher condensing temperatures, a vapor compression cycle
Fig. 12 shows the efficiency improvement of the power plant as would be required.
a function of the datacenter cycle’s condensing temperature. The In terms of CO2 footprint, Fig. 13 shows the reduction in the
figure shows that the higher the condensing temperature, the amount of CO2 per kilowatt-hour output per year as a function of
greater the efficiency improvements. The efficiency of the plant can the datacenter condensing temperature. Also shown on the graph is
be improved by up to 2.2% if the datacenter’s waste heat is reused in the amount of CO2 saved per kilowatt-hour output per year. These
values assume that coal is used as the source of the power plant’s
energy and will be less for other types of fuel. Therefore, if a power
1000 130 plant with an output capacity of 500 MW were considered, the
875.5

Heat exchanger volume savings in CO2 would be in the order of 195 000 tons of CO2 per
Heat exchanger volume [cm 3 ]

Secondary fluid mass flow rate year. This could potentially save about $ 3 000 000 per year if
Mass flow rate [kg/h]

850 100
a carbon tax of $15 per ton of CO2 [23] was considered.

700 70
729.9

Table 8
705.2

704.0

Operating conditions for the power utility and datacenter cooling cycle.

550 40 Power utility Datacenter cooling


631.6

Boiler Evaporator
Pressure [MPa] 16.55 1.681
400 10 Turbine Compressor
HFC134a HFC134a HFO1234ze Water HFC134a Inlet Pressure [MPa] 16.55 1.681
Case1 Case2 Case3 Case4 Case5 Inlet Temperature 
[C] 538 70
LP LP LP LP VC
Condenser
Condensing Temperature 
[C] 46 60e100
Fig. 9. Heat exchanger volume and secondary fluid mass flow rate.

Please cite this article in press as: J.B. Marcinichen, et al., On-chip two-phase cooling of datacenters: Cooling system and energy recovery
evaluation, Applied Thermal Engineering (2012), doi:10.1016/j.applthermaleng.2011.12.008
J.B. Marcinichen et al. / Applied Thermal Engineering xxx (2012) 1e16 11

5.2. Datacenter
0.479
0.48
Due to datacenter growth, it is more than likely to see data-
Thermal Efficiency, -

centers in excess of 100 000 servers (viz. Fig. 14). Therefore, the
simulations following will be based on a datacenter containing
0.47
100 000 servers, with each server generating 325 W, which
0.462 includes the main processor and auxiliary electronics (memories,
DCeDC converters, hard drives, etc). Fig. 15 shows a graph of the
0.46
total power supply required by the datacenter to operate the IT
equipment and the cooling system. Included in the graph is on-chip
two-phase cooling methods using a vapor compression cycle and
0.45 liquid pumping cycle. As a comparison, traditional air cooling units
40 50 60 70 80 90 100 are also simulated, where it is assumed that their power
Condensing Temperature, oC consumption is the same as that required to operate the IT equip-
ment ([22] and [21]). The results are plotted as a function of the
Fig. 12. Thermal efficiency of power plant.
compressor or pump overall efficiencies (defined as the ratio
between isentropic compression or pumping power and the elec-
trical power for the drivers). All simulations were developed for an
0.97 0.05 evaporating temperature of 60  C (on the chip), with condensing
temperatures being 60  C and 90  C for the liquid pumping and
kg CO2 savings / kWh
0.96 0.04
vapor compression cycles, respectively.
0.95 0.03 It is seen that the cycle using a vapor compressor has a strong
kg CO2 / kWh

function of the compressor’s overall efficiency, up to a value of


0.94 0.02 approximately 35% after which it becomes less dependent. Typi-
cally, compressors have an overall efficiency of about 60%. The
0.93 0.01 liquid pumping cycle hardly shows any dependence on the pump
efficiency. This is due to the power required to drive the pump
0.92 0.00
being very low [28]. The higher power consumption of the
0.91 compressor is due to the energy required to increase the pressure
40 50 60 70 80 90 100 from a saturation temperature of 60  C to 90  C. What is noticed,
Condensing temperature, oC though, is that the datacenter’s power requirements are reduced
considerably for overall efficiencies above 15% (vapor compression
Fig. 13. CO2 footprint and savings per kilowatt-hour. cycle) when compared to traditional air cooling. This reduction is in

Fig. 14. Datacenter size and information technology (IT) power requirements.

Please cite this article in press as: J.B. Marcinichen, et al., On-chip two-phase cooling of datacenters: Cooling system and energy recovery
evaluation, Applied Thermal Engineering (2012), doi:10.1016/j.applthermaleng.2011.12.008
12 J.B. Marcinichen et al. / Applied Thermal Engineering xxx (2012) 1e16

Air cooling + IT the case for a liquid pumping cycle due to its energy consumption
Vapor compressor + IT being very low. The vapor compression cycle always has some
80
unrecoverable heat due to inefficiencies, with this heat being lost to
Data Center Power Supply, MW

Liquid pump + IT
70 the environment. At this point in stage it would appear as if the use
IT load
of a liquid pumping cycle far outweighs the vapor compression
60 cycle. However, what the graphs do not show is the quality of the
heat being sold. The former can sell heat at a temperature of 60  C,
50 while the latter can sell it at 90  C. The quality of heat is therefore
important, not only due to the monetary value it adds, but also to
40 the application to which it is sold. A limited number of applications
can use 60  C of heat, with the limits becoming less as the
30 temperature is increased.

20
0 20 40 60 80 100 5.3. Exergy analysis
Compressor and liquid pump overall efficiency, %
To better explore and understand the difference between the
Fig. 15. Datacenter power supply - IT and cooling system.
two possibilities of cooling cycles (vapor compressor and liquid
pump) regarding energy recovery, i.e., exergy available at the
the order of 50% when using a liquid pumping cycle and 41% for condenser for a secondary application, the concept of exergy is
a vapor compression cycle with a compressor having an overall introduced. The cooling cycles defined in the previous section were
efficiency of 60%. considered (viz. Figs. 3 and 4). For this analysis the heat load of an
Further savings can be made when energy recovery is consid- entire blade center composed of 14 blades, which has the micro-
ered. Since on-chip cooling is used, recovering this energy would be processors and auxiliary electronics to be cooled, is defined as
a simple process by just incorporating a condenser, where the a heat source of 4900 W at 85  C, while a counter-flow heat
energy absorbed by the fluid from the server is transferred to exchanger is used as the condenser. The secondary fluid is water at
a secondary fluid, like water, in the condenser. Fig. 16 shows a graph an inlet and outlet temperature of 46  C and 5  C lower than the
of the datacenter power supply for the three types of cooling inlet temperature of working fluid, respectively. [38] showed for
technologies as a function of the efficiency with which energy is energy recovery applications, that 46  C is about the condenser
recovered. 100% efficiency implies that all the heat generated by the outlet temperature of a coal power plant, being the potential point
servers is recovered, while 0% means that none of the heat is where the recovered heat of a datacenter can be injected. An overall
recovered. Note that there is no change in power consumption for efficiency of 60% was considered for the drivers, i.e., pump and
air cooling as it was assumed that the heat was not recovered, compressor. HFC134a is used as working fluid and a MEþMPAE
although this technology does exist, albeit not as effective as for evaporating temperature of 60  C is defined. An outlet vapor quality
liquid cooling. The plots for the liquid pumping and vapor of about 80% at the MPAE is obtained from the simulations of both
compression cycles assume that the pump and compressor has an cycles. The mass flow rate of the working fluid and secondary fluid
overall efficiency of 100% and 60%, respectively, although the choice were adjusted so that the energy balance is respected.
of efficiency for the pump is negligible (viz. Fig. 15). It should be The steady state exergy rate balance is defined by Eq. (2) [35].
noted that 0 MW datacenter supply does not mean the datacenter The first and second terms in the right side of equality represent the
requires no power, but rather that all the power received as elec- exergy transfer accompanying heat and work, the third and fourth
tricity is sold as heat. The financial implications would show this are the time rate of exergy transfer accompanying mass flow and
since the value of the heat sold would be different from the elec- flow work and, finally, the last term is the rate of exergy destroyed.
tricity purchased and will be a function on the application to which With some mathematical manipulation it is possible to prove that
the heat is sold. the last term in Eq. (2), i.e., E_ d is the entropy generation multiplied
Fig. 16 shows that practically all the power purchased in the by the dead state temperature T0.
form of electricity can potentially be sold as heat. This is especially !
X T0 _ X X
0 ¼ 1 Qj  W _ cv þ m _ i efi  m_ e efe
Tj e
j i
70 |fflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflffl{zfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflffl}
rate of exergy transfer
Data Center Power Supply, MW

60
Air cooling + IT  E_ d (2)
Vapor compressor + IT
|{z}
50 rate of exergy destruction
Liquid pump + IT
40 IT load It can be observed that an exergy reference environment is
necessary to be defined. Such an environment represents the state
30
of equilibrium or dead state. This equilibrium state defines the
20 exergy as the maximum theoretical work obtainable when another
system in a non-equilibrium state interacts with the environment
10 to the equilibrium. For the present work, the reference is defined as
295 K, 100 kPa for water and 295 K, 603.28 kPa and 50% of vapor
0 quality for HFC134a.
0 20 40 60 80 100
The goal of the analysis is to determine, for each cycle, the
Energy Recovery Efficiency, %
exergy supplied, recovered and destroyed for a control volume in
Fig. 16. Datacenter power supply - IT and cooling system - potential of energy the cooling cycle. With this the overall exergetic efficiency, defined
recovery. as the ratio between recovered and supplied exergies, can be

Please cite this article in press as: J.B. Marcinichen, et al., On-chip two-phase cooling of datacenters: Cooling system and energy recovery
evaluation, Applied Thermal Engineering (2012), doi:10.1016/j.applthermaleng.2011.12.008
J.B. Marcinichen et al. / Applied Thermal Engineering xxx (2012) 1e16 13

determined. The exergetic efficiency of each component is also 0.60


evaluated. It qualitatively identifies and classifies the components
VC cooling cycle

Exergetic efficiency, [-]


that present higher irreversibilities, helping to decide which 0.55
component to optimize to improve the thermodynamic perfor- LP cooling cycle
mance of the cooling cycle. 0.50
Table 9 shows the results obtained regarding exergy and energy.
For this analysis it is worth mentioning that the piping connecting
0.45
the main components is considered to be adiabatic having no
pressure losses, while the potential and kinetic energy effects in the
0.40
cooling cycles are also neglected.
First, it can be seen that the total exergy recovered is higher for
the VC cooling cycle, which is a consequence of the higher exergy 0.35
0.4 0.5 0.6 0.7 0.8 0.9 1
supplied by the driver used in this cycle. It is highlighted here that
this high exergy is the subject of interest of the owner of a coal fired Driver overall efficiency, [-]
power plant presented beforehand, as will be discussed in the
Fig. 17. Exergetic efficiency versus driver overall efficiency.
following sections.
Of the components considered in the cooling cycles, the driver
showed the lowest exergetic efficiency, which implies that to
showed beforehand. Exergy analysis clearly identifies efficiency
improve the thermodynamic performance of the cooling cycles the
improvements and reductions in thermodynamics losses attribut-
first component to be optimized in the design would be the driver.
able to green technologies. Additional advantages of such analysis
It is important to mention that since the simulations considered
are the potential to evaluate green technology aspects such as
a fixed driver overall efficiency and the heat losses and pressure
environmental impact or sustainable development (normally
drops in the piping were neglected, means that such an analysis is
associated with carbon dioxide emissions) and economics (“exergy,
limited to the main components of the cycle and a complete
not energy, is the commodity of value in a system, and assign costs
simulation must be done to generalize the conclusions.
and/or prices to exergy-related variables”) [46].
It can also be observed that the overall exergetic efficiency is lower
for the VC cooling cycle, with the compressor and ME þ MPAE being
the main culprits. The overall exergetic efficiency also shows that 5.4. Carbon footprint
there is a huge need to improve the thermodynamic performance of
the cooling cycle, since less than 50% of the available exergy is used. For the calculation of the carbon footprint, only the contribution
Fig. 17 shows the exergetic efficiency as a function of the driver of the electricity used is considered. The effect of greenhouse gases
overall efficiency. It can be seen that the exergetic efficiency (GHG) being formed by the manufacturing, transporting, storage
increases when the overall efficiency increases and the VC cooling and disposal of the components of the datacenter, as well as the
cycle starts to be the best exergetic option after a driver overall datacenter building, fall under the Life Cycle Assessment, which
efficiency of about 67%. It can be concluded that for a proper falls outside the scope of the current paper. Further, of the green-
utilization of exergy the VC cooling cycle must have an overall house gases, only CO2 will be considered as it contributes to more
efficiency higher than 67%. Only in this way such a cycle is than 75% of all the greenhouse gases and contributes the most to
competitive with the LP cooling cycle, which showed low effect of the greenhouse effect. Fig. 18 shows the reduction of the quantity of
the driver overall efficiency on the exergetic efficiency. CO2 for the three cooling technologies as a function of the efficiency
Finally, it is important mentioning that the thermodynamic with which the energy is recovered. The quantity of CO2 is calcu-
performance alone (energy balance) does not permit the analysis lated with the assumption that the datacenter purchases its elec-
tricity from a power plant running on coal and that it is selling
waste heat back to the power plant, as discussed earlier. This graph,
Table 9 therefore, takes into consideration the efficiency increase of the
Exergy and energy balance results. power plant, since the amount of CO2 released is a function of the
power plant’s efficiency, which in itself is a function of the effi-
Energy balance
ciency with which energy is recovered.
Energy in LP cycle VC cycle Energy out LP cycle VC cycle
Pump or compressor 35.49 1126 Heat out in the 4921.29 5575.5
input power [W] condenser [W]
Heat added in the 4900 4900 600
ME þ MPAE [W]
Exergy balance 500
Mass CO2, 106 kg per year

Air cooling + IT
LP cycle VC cycle LP cycle VC cycle
Exergy supplied [W] 900.8 1991 Exergy 428.1 890 400 Compressor + IT
recovered [W] Liquid pump + IT
300
Exergy destroyed or Exergetic
irreversibility [W] efficiency [%]
200
Pump or compressor 14.2 450.4 Pump or 60.0 60.0
compressor 100
Condenser 137.6 209.2 Condenser 75.7 81.0
ME þ MPAE 320.9 320.8 ME þ MPAE 62.3 62.3 0
iHEx e 39.12 iHEx e 82.8 0 0.2 0.4 0.6 0.8 1
LPR e 25.86 LPR e 82.3
Energy recovery efficiency, -
EEV e 55.79 EEV e 0
Total 472.7 1101 Overall 47.5 44.7
Fig. 18. Reduction in CO2 for three datacenter cooling technologies.

Please cite this article in press as: J.B. Marcinichen, et al., On-chip two-phase cooling of datacenters: Cooling system and energy recovery
evaluation, Applied Thermal Engineering (2012), doi:10.1016/j.applthermaleng.2011.12.008
14 J.B. Marcinichen et al. / Applied Thermal Engineering xxx (2012) 1e16

The figure shows that the datacenter could potentially have


x 106
a zero carbon footprint regarding electricity usage when on-chip 350
cooling using a liquid pumping or vapor compression cycle is used. 300

Datacenter CO2 footprint,


The use of a liquid pumping or vapor compression cooling cycle Datacenter - Compressor
250 Datacenter - Liquid pump
without energy recovery (0% recovery efficiency), compared to

kg CO2 per year


200
traditional air cooling, reduces the carbon footprint of the datacenter
considerably, with a reduction of 50% for the former and 40% for the 150
latter. This reduction is improved further with energy recovery, with 100
a potential reduction of almost 100% and 96% being achievable. 50
As observed previously, Fig. 18 does not show the quality of the
0
heat being recovered, with the value and applicability of this heat
-50
not being clear. This can be seen more clearly when applying the
waste heat to the thermal power plant, which is shown in Fig. 19. -100
40 50 60 70 80 90 100
This graph shows the CO2 reduction of the datacenter due to energy
recovery and the savings in CO2 of the power plant due to efficiency Condensing temperature, oC
improvements. Instead of plotting the carbon footprint as a func-
Fig. 20. CO2 footprint of a datacenter considering CO2 reduction of power plant.
tion of the recovery efficiency it is plotted as a function of the
condenser temperature, which is directly linked to the feedwater
heater of the power plant. The effect is the same since a lower 5.5. Monetary savings
energy recovery efficiency would result in lower temperature
increases of the power plant’s feedwater and, hence, lower effi- Global warming is having a huge impact on the environment
ciency increases. The graph therefore shows the limit of each and on the livelihood of people regarding food production and
cooling cycle. Since the power plant’s thermal efficiency improves natural resources. To counter this, a carbon tax is being introduced,
with increase in condenser temperature, the amount of CO2 saved which is aimed at helping the environment by not only reducing
by the power plant increases. However, when using a liquid carbon emissions by forcing people and organizations to become
pumping cycle for the datacenter, only 25% of the total potential more energy-efficient, but also by raising funds to be used for clean
savings in CO2 can be achieved, amounting to approximately 17 000 energy research. The tax is levied on the carbon content of fuels,
tons per year for a 173 MW power plant. By making use of a vapor increasing the competitiveness of non-carbon technologies such as
compression cycle, however, the potential savings in CO2 can reach solar, wind or nuclear energy sources. Therefore, organizations
as high as 70 000 tons per year. Therefore, although the liquid using electricity produced from the burning of fossil fuels will pay
pumping cycle was the better performing cooling cycle regarding a higher tax than those produced from non-carbon burning fuels.
energy usage and CO2 reduction, due to the higher temperatures The probability also exists for taxing the utility generating the
achievable the vapor compression cycle has a larger impact on the electricity. Carbon taxes have only been introduced in a few
secondary application making use of the waste heat. countries, with most European countries taking the lead, even
Of course, depending on the point of view taken, it may be though the way organizations are being taxed vary from country to
argued that due to the savings the power plant is making in CO2 due country. In the United States the introduction of carbon tax has
to the datacenter, the datacenter could claim those savings as part been made in California and the city of Boulder, Colorado, with
of its own reduction. This could potentially then lead to the data- taxes being in the order of 4 cents/ton of CO2. European countries
center having a negative carbon footprint regarding energy usage, have been much more stringent with taxes in some countries, such
which it can then use as a carbon offset, as shown in Fig. 20. This as Sweden, being as high as $100 per ton of CO2 [8]. The Larson Bill
offset could be used to compensate for other CO2 emitting [23] proposes to introduce a nationwide tax (US) of $15/ton CO2
processes in the datacenter, or can even be sold to other organi- starting in 2012, increasing by $10/ton CO2 every year. It also
zations as is done in carbon-trading. This should, however, be proposes to increase this increment to $15/ton CO2 after 5 years if
viewed as a tentative idea as regulations would determine whether the US emissions stray from the Environmental Protection Agency’s
this is possible or not. However, if this were the case, by making use (EPA) glide-path prediction, which proposes to cut emission to 80%
of a vapor compression cycle, the offset could be 167% more than that of 2005 levels by 2050.
when using a liquid pumping cycle. With a recommended price of $30/ton CO2 [37] could cost
industries millions if efficiencies are not improved. Datacenters are

Datacenter - Compressor
350 70
Datacenter - Liquid pump
Mass CO2 consumed (datacenter),

50 4.5
Mass CO2 savings (power plant),

300 Power plant 60


45 3.6
250 50
Datacenter savings,

Power plant savings,


106 kg per year

106 kg per year

M$ per year

200 40 40 2.7
M$ per year

150 30
35 1.8
100 20 Datacenter - Compressor
30 0.9
50 10 Datacenter - Liquid pump
Power plant
0 0 25 0
40 60 80 100 40 50 60 70 80 90 100
Condensing temperature, oC Condensing temperature, oC

Fig. 19. Carbon footprint of datacenter and CO2 savings of power plant. Fig. 21. Datacenter and power plant savings.

Please cite this article in press as: J.B. Marcinichen, et al., On-chip two-phase cooling of datacenters: Cooling system and energy recovery
evaluation, Applied Thermal Engineering (2012), doi:10.1016/j.applthermaleng.2011.12.008
J.B. Marcinichen et al. / Applied Thermal Engineering xxx (2012) 1e16 15

50 An exergy analyses of the cooling cycles, regarding the potential


Liquid pump of exergy recovery at the condenser, showed a low overall exergetic
45 efficiency (lower than 50%), meaning that improvements can be
Compressor
done to increase the thermodynamic performance of the cycles.
60o C
When looking at local effects, such analyses showed that the driver
Total savings,
M$ per year

40
and the MEþMPAE are the components with the lowest exergetic
90o C efficiency and would be the main components to be improved in
35 terms of thermodynamic design. It was also shown that the overall
exergetic efficiency of the vapor compression cooling cycle is
30 strongly influenced by the compressor overall efficiency, which
showed to be more exergeticaly efficient than the liquid pumping
cooling cycle for an overall efficiency higher than 67%.
25
0 0.2 0.4 0.6 0.8 1 A case study was developed to investigate the potential savings
in energy a datacenter can make by implementing on-chip cooling
Energy recovery efficiency, -
with waste heat recovery. As an application for the waste heat,
Fig. 22. Total savings for liquid pumping and vapor compression cooling technologies. a coal fired power plant was analyzed. The results showed that,
when compared with traditional air cooling systems, the energy
consumption of the datacenter could be reduced by as much as 50%
also not exempt from these taxes, which will be introduced in the when using a liquid pumping cycle and 41% when using a vapor
following years to come [33]. Fig. 21 shows the potential savings compression cycle. The overall consumption can be reduced even
made by a datacenter with a size of 100 000 servers and a coal further if the recovered energy is sold to a secondary application,
power plant with a size of 175 MW if heat were captured from the such as a thermal power plant. Power plant thermal efficiency
datacenter and sold to the power plant. The savings not only improvements in the order of 2.2% are possible if datacenter waste
includes that saved in energy costs by implementing a liquid heat is incorporated in the power plant’s feedwater. This could
pumping or vapor compression cycle instead of a traditional air imply huge savings in terms of fuel as well as carbon tax due to
cooling cycle, but also that saved in carbon tax. The savings of the a reduced carbon footprint.
power plant is in terms of fuel saved and the savings made in
carbon tax. For fuel costs, a value of $90/ton of coal was used.
Acknowledgements
For the datacenter, the graph shows that most savings are made
if a liquid pumping cycle was used, with the potential savings being
The Commission for Technology and Innovation (CTI) contract
in the order of $45 000 000 per year, while a vapor compression
number 6862.2 DCS-NM entitled “Micro-Evaporation Cooling
cycle would save in the order of $40 000 000 per year. The power
System for High Performance Micro-Processors: Development of
plant, when recovering heat from a datacenter using a liquid
Prototype Units and Performance Testing” directed by the LTCM
pumping cycle, will only save about $1 000 000 per year, with
laboratory sponsored this work along with the project’s industrial
savings reaching almost $4 500 000 a year if a vapor compression
partners: IBM Zürich Research Laboratory (Switzerland) and
cycle were used.
Embraco (Brazil). J.B. Marcinichen wishes to thank CAPES (“Coor-
The overall savings are given in Fig. 22. This graph shows that
denação de Aperfeiçoamento de Pessoal de Nível Superior”) for
the total savings, if a datacenter was to sell waste heat at 60  C with
a one year fellowship to work at the LTCM laboratory. The authors
a liquid pumping cycle, could be about $46 000 000 a year, while
also wish to thank Honeywell Inc. for providing the thermophysical
selling heat at 90  C with a vapor compression cycle could save
properties of HFO1234ze for the calculations.
a total of $43 000 000 per year. These are savings that a customer
would potentially not have to pay. Even though it seems as if the
clear solution is to use a liquid pumping cycle within a datacenter, References
the incentive for a power plant to cooperate with a datacenter
[1] B. Agostini, M. Fabbri, J.E. Park, L. Wojtan, J.R. Thome, B. Michel, State of the art
would be greater if a vapor compression cycle was used. of high heat flux cooling technologies, Heat Transfer Engineering 28 (2007)
258e281.
6. Conclusion [2] B. Agostini, M. Fabbri, J.R. Thome, B. Michel, High heat flux two-phase cooling
in silicon multimicrochannels, IEEE Transactions on Components and Pack-
aging Technologies 31 (3) (2008) 691e701.
A hybrid two-phase cooling cycle has been proposed and [3] A. Azzi, U.S.T.H.B. Alger, L. Friedel, Two-phase upward flow 90o bend pressure
simulated to cool microprocessors and auxiliary electronics of loss model, Forschung im Ingenieurwesen 69 (2005) 120e130.
[4] A. Azzi, L. Friedel, S. Belaadi, Two-phase gas/liquid flow pressure loss in bends,
blade server boards with two-phase evaporating flow in the micro- Forschung im Ingenieurwesen 65 (2000) 309e318.
evaporator cooling elements. A simulation code was developed and [5] A. Barnea, Transition from annular flow and from dispersed bubble flow e
5 cases were simulated considering 3 different working fluids, unified models for the whole range of pipe inclinations, International Journal
of Multiphase Flow 12 (5) (1986) 733e744.
HFC134a, HFO1234ze and water (in an analogous single-phase [6] D. Barnea, O. Shoham, Y. Taitel, Flow pattern transition for vertical downward
cooling cycle) and different internal diameters of the pipes and two phase flow, Chemical Engineering Science 37 (5) (1982) 741e744.
elbows joining the components. The results showed that the liquid [7] H. Blasius, Das Ähnlichkeitsgesetz bei Reibungsvorgängen in Flussigkeiten
(1913) Fors-chg. Arb. Ing. -Wes., 131 Berlin.
water cooling cycle has a pumping power consumption 5.5 times [8] Carbon Tax, In Wikipedia, the Free Encyclopedia Retrieved 08:12, July 26,
that obtained for the two-phase HFC134a cooling cycle, both 2010, (2010) Available from: http://en.wikipedia.org/w/index.php?
considering a liquid pump as the driver of the fluid. When title¼Carbon_tax&oldid¼374709823 :.
[9] A. Cavallini, D. Del Col, L. Doretti, G.A. Longo, L. Rossetto, Heat transfer and
compared with the HFO1234ze cooling cycle the difference drops to
pressure drop during condensation of refrigerants inside horizontal enhanced
4.4 times. The simulation of the vapor compression cooling cycle tubes, International Journal of Refrigeration 23 (2000) 4e25.
showed higher pumping power consumption when compared with [10] A. Cicchitti, C. Lombardi, M. Silvestri, G. Soldaini, R. Zavattarelli, Two-phase
the other cycles simulated. However, this cycle can be justified cooling experiments-pressure drop, heat transfer and burnout measurements,
Energia Nuclear 7 (1960) 407e425.
when the waste heat at the condenser is recovered for applications [13] F.W. Dittus, L.M.K. Boelter, Publications on Engineering, vol. 2, University of
such as district heating and preheating of boiler feedwater. California, Berkeley, 1930, 443.

Please cite this article in press as: J.B. Marcinichen, et al., On-chip two-phase cooling of datacenters: Cooling system and energy recovery
evaluation, Applied Thermal Engineering (2012), doi:10.1016/j.applthermaleng.2011.12.008
16 J.B. Marcinichen et al. / Applied Thermal Engineering xxx (2012) 1e16

[14] V. Dupont, J.R. Thome, A.M. Jacobi, Heat transfer model for evaporation in [37] W. Nordhaus, A Question of Balance e Weighing the Options of Global
microchannels. Part II: comparison with the database, International Journal of Warming Policies, Yale University Press, New Haven, 2008.
Heat and Mass Transfer 47 (2004) 3387e3401. [38] Olivier, J.A., Marcinichen, J.B and Thome, J.R., 2010, “Two-phase cooling of
[15] EPA, Report to Congress on Server and Data Center Energy Efficiency Public datacenters: reduction in energy costs and improved efficiencies”, in
Law pp. 109e431, U.S. Environmental Protection Agency, 2007. Proceedings of the 13th Brazilian congress of thermal sciences and engi-
[16] P. Ganapati, Water-Cooled Supercomputer Doubles as Dorm Space Heater neering e ENCIT2010, Uberlandia, MG, Brazil.
Retrieved 12:12, February 18, 2011, (2009) Available from: http://www. [39] C.L. Ong, J.R. Thome, Macro-to-microchannel transition in two-phase flow:
wired.com/gadgetlab/2009/06/ibm-supercomputer/. part 2 - flow boiling heat transfer and critical heat flux, Experimental Thermal
[17] J.M. Gonçalves, C. Melo, C.J.L. Hermes, J.R. Barbosa, Experimental mapping of the and Fluid Science (2011). doi:10.1016/j.expthermflusci.2010.12.003.
thermodynamic losses in vapor compression refrigeration systems, Journal of the [40] Park, J.E., 2008. “Critical heat flux in multi-microchannel copper elements
Brazilian Society of Mechanical Sciences and Engineering 33 (2) (2011) 159e165. with low pressure refrigerants”, PhD Thesis, École Polytechnique Fédérale de
[18] R. Hannemann, J. Marsala, M. Pitasi, Pumped liquid multiphase cooling, in: Lausanne, Switzerland.
Proceedings IMECE - International Mechanical Engineering Congress and [41] J.E. Park, J.R. Thome, Critical heat flux in multi-microchannel copper elements
Exposition, Anaheim, CA, USA, Paper 60669 (2004). with low pressure refrigerants, International Journal of Heat and Mass
[19] C.J.L. Hermes, C. Melo, F.T. Knabben, J.M. Gonçalves, Prediction of the energy Transfer 53 (2010) 110e122.
consumption of household refrigerators and freezers via steady-state simu- [42] Perez-Tellez, C., 2003, “Improved bottomhole pressure control for
lation, International Journal of Applied Energy 86 (2009) 1311e1319. underbalanced drilling operations”, Ph.D. Dissertation, Louisiana State
[20] IBM, IBM BladeCenter: Build Smarter IT Retrieved 09:30, May 20, 2011, (2010) University.
Available from: ftp://public.dhe.ibm.com/common/ssi/ecm/en/blb03002usen/ [43] T.S. Ravigururajan, A.E. Bergles, General correlations for pressure
BLB03002USEN.PDF. drop and heat transfer for single-phase turbulent flow in internally
[21] J. Ishimine, Y. Ohba, S. Ikeda, M. Suzuki, Improving IDC cooling and air condi- ribbed tubes, Augmentation of Heat Transfer in Energy Systems 52
tioning efficiency, FUJITSU Scientific & Technical Journal 45 (2009) 123e133. (1985) 9e20.
[22] J.G. Koomey, Estimating Total Power Consumption by Servers in the U.S. and [44] R. Revellin, J.R. Thome, A theoretical model for the prediction of the critical
the World February 15,, Analytics Press, Oakland, CA, 2007, Available from: heat flux in heated microchannels, International Journal of Heat and Mass
http://enterprise.amd.com/us-en/AMD-Business/Technology-Home/Power- Transfer 51 (2008) 1216e1225.
Management.aspx. [45] G. Ribatski, L. Wojtan, J.R. Thome, An analysis of experimental data and
[23] J.B. Larson, America’s Energy Security Trust Fund Act of 2009 (2009) H.R. 1337. prediction methods for two-phase frictional pressure drop and flow boiling
[24] J. Lee, I. Mudawar, Low-Temperature two-phase microchannel cooling for heat transfer in micro-scale channels, Experimental Thermal and Fluid
high-Heat-Flux thermal Management of Defense electronics, IEEE Trans- Science 31 (2006) 1e19.
actions on Components and Packaging Technologies 32 (2) (2009) 453e465. [46] M.A. Rosen, I. Dincer, M. Kanoglu, Role of exergy in increasing efficiency and
[25] P.L. Leonard, A.L. Phillips, The Thermal Bus Opportunity e A Quantum Leap in sustainability and reducing environmental impact, Energy Policy 36 (2008)
Data Center Cooling Potential, ASHRAE Transactions, Denver, CO, 2005. 128e137.
[26] D. Liu, S. Wang, Flow pattern and pressure drop of upward two-phase flow in [47] M. Saini, R.L. Webb, Heat rejection limits of air cooled plane fin heat sinks for
vertical capillaries, Industrial and Engineering Chemistry Research 47 (2008) computer cooling, IEEE Transactions on Components and Packaging Tech-
243e255. nologies 26 (1) (2003) 71e79.
[27] Y. Madhour, J.A. Olivier, E. Costa-Patry, S. Paredes, B. Michel, J.R. Thome, Flow [48] E. Samadiani, S. Joshi, F. Mistree, The thermal design of a next generation data
Boiling of R134a in a Multi-Microchannel Heat Sink with Hotspot Heaters for center: a conceptual exposition, Journal of Electronic Packing 130 (2008)
Energy-Efficient Microelectronic CPU Cooling Applications, IEEE Transactions 041104e41111 041104e041108.
on Components, Packaging and Manufacturing Technology 1 (6) (2011) [49] R.R. Schmidt, B.D. Notohardjono, High-end server low-temperature cooling,
873e883. IBM Journal of Research and Development 46 (2002) 739e751.
[28] Marcinichen, J.B., Thome, J.R., 2010, “New Novel Green Computer Two-Phase [50] P.L. Spedding, E. Benard, G.M. McNally, Fluid flow through 90 degree bends,
Cooling Cycle: a Model for Its Steady-State Simulation”, in Proceedings of the Developments in Chemical Engineering and Mineral Processing 12 (2004)
23rd International Conference on Efficiency, Cost, Optimization, Simulation and 107e128.
Environmental Impact of Energy Systems - ECOS2010, Lausanne, Switzerland. [51] Y. Taitel, D. Bornea, A.E. Dukler, Modelling flow pattern transitions for steady
[29] J.B. Marcinichen, J.R. Thome, B. Michel, Cooling of microprocessors with upward gas-liquid flow in vertical tubes, AIChE Journal 26 (3) (1980)
micro-evaporation: a novel two-phase cooling cycle, International Journal of 345e355.
Refrigeration (2010). [52] Thome, J.R., Agostini, B., Revellin, R. and Park, J.E., 2007, “Recent advances in
[30] J.B. Marcinichen, J.A. Olivier, V. Oliveira, J.R. Thome, A Review of On-Chip thermal modeling of micro-evaporators for cooling of microprocessors”, in
Micro-Evaporation: Experimental Evaluation of Liquid Pumping and Vapor Proceedings of the ASME International Mechanical engineering Congress and
Compression Driven Cooling Systems and Control, International Journal of exposition (IMECE), Seattle, Washington.
Applied Energy 92 (2011) 147e161. [53] Thome, J.R. and Bruch, A., 2008, “Refrigerated cooling of microprocessors with
[31] A.W. Mauro, J.R. Thome, D. Toto, G.P. Vanoli, Saturated critical heat flux in micro-evaporation heat sinks: new development and energy Conservation
a multi-microchannel heat sink Fed by a split flow system, Experimental Prospects for green datacenters”, in Proceedings Institute of refrigeration
Thermal and Fluid Science 34 (2010) 81e92. (IOR).
[32] J.P. Meyer, J.A. Olivier, Transitional flow inside enhanced tubes for fully devel- [54] J. Thome, V. Dupont, A. Jacobi, Heat transfer model for evaporation in
oped and developing flow with different types of inlet disturbances: part IIeheat microchannels. Part I: presentation of the model, International Journal of Heat
transfer, International Journal of Heat and Mass Transfer 54 (2011) 1598e1607. and Mass Transfer 47 (2004) 3375e3385.
[33] R.L. Mitchel, Carbon tax could whack data centers, Computerworld (2010) [55] S. Trutassanawin, E.A. Groll, S.V. Garimella, L. Cremaschi, Experimental
Available from: http://blogs.computerworld.com/16128/carbon_tax_could_ investigation of a miniature-scale refrigeration system for electronics cooling,
whack_data_centers\. IEEE Transactions on Components and Packaging Technologies 29 (3) (2006)
[34] Mongia, R., Masahiro, K., DiStefano, E., Barry, J., Chen, W., Izenson, M., Pos- 678e687.
samai, F., Zimmermann, A. and Mochizuki, M., 2006, “Small scale refrigeration [56] S. Voss, G. Gould, The Rankine Cycle: Workhorse of the Coal-fired Utility
system for electronics cooling within a notebook computer”, in Proceedings Industry, TechBriefs, Burns & McDonnell, 2001, No 3, pp. 4e6.
ITHERM, San Diego, CA. [57] World_Coal_Association, Coal & Electricity (2011) Available from: http://
[35] M.J. Moran, I. Howard, N. Shapiro, Fundamentals of Engineering Thermody- www.worldcoal.org /coal/uses-of-coal/coal-electricity/.
namics, sixth ed. John Wiley & Sons, 2010, 725 pages. [58] R. Zhou, T. Zhang, J. Catano, J.T. Wen, G.J. Michna, Y. Peles, M.K. Jensen, The
[36] H. Muller-Steinhagen, K. Heck, A simple pressure drop correlation for two- steady-state modeling and optimization of a refrigeration system for high
phase flow in pipes, Chemical Engineering and Processing: Process Intensifi- heat flux removal, Applied Thermal Engineering (2010). doi:10.1016/
cation 20 (1986) 297e308. j.applthermaleng.2010.05.023.

Please cite this article in press as: J.B. Marcinichen, et al., On-chip two-phase cooling of datacenters: Cooling system and energy recovery
evaluation, Applied Thermal Engineering (2012), doi:10.1016/j.applthermaleng.2011.12.008

You might also like