See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/305525405

Effects of mineral oil immersion cooling on IT equipment reliability and

reliability enhancements to data center operations

Conference Paper · May 2016

DOI: 10.1109/ITHERM.2016.7517566

CITATIONS READS

49 11,040

4 authors:

Jimil M. Shah Rick Eiland

TMGcore University of Texas at Arlington
39 PUBLICATIONS 323 CITATIONS 13 PUBLICATIONS 163 CITATIONS

SEE PROFILE SEE PROFILE

Ashwin Siddarth Dereje Agonafer

17 PUBLICATIONS 146 CITATIONS

University of Texas at Arlington
290 PUBLICATIONS 1,791 CITATIONS
SEE PROFILE
SEE PROFILE

All content following this page was uploaded by Jimil M. Shah on 16 April 2018.

The user has requested enhancement of the downloaded file.

 Effects Of Mineral Oil Immersion Cooling On IT
Equipment Reliability And Reliability Enhancements
To Data Center Operations
Jimil M. Shah, Richard Eiland, Ashwin Siddarth and Dereje Agonafer
University of Texas at Arlington
Arlington, TX, United States, 76013
Email: [email protected]

Abstract— This paper reviews the changes in physical and Keywords— information technology equipment; reliability;
chemical properties of information technology (IT) equipment compatibility; reliability enhancements
materials like polyvinyl chloride (PVC), printed circuit board
(PCB) and switching devices due to mineral oil to characterize
the interconnect reliability of materials. By submerging all of a I. INTRODUCTION
server’s heat-generating components in a dielectric liquid, creates Submerging servers and IT equipment in a mineral oil, a
the attack on reliability issues at the device level. The improved dielectric liquid, for cooling purposes, enables substantial
efficiency of mineral oil may offer simplicity in facility design
energy savings today and accommodates growing load
compared to traditional air cooling and provide a means for cost
savings. In spite of its improved cooling efficiency and cost densities of future facilities. In the current scenario, the lack of
savings, a mineral oil immersion cooling technique is still not extremely high load densities that would make traditional air
widely implemented and suppliers are reluctant to jeopardize cooling inadequate has forced the discussion to focus on just
sales of existing air-based cooling system equipment. Only the efficiency merits of this technology. The existing
compelling physics regarding direct immersion cooling is not proprietary submersion cooling solutions [1], [3], [4] and
enough for data center operators. Many uncertainties and numerous case studies [2], [3] have established the
concerns persist regarding the effects of mineral oil immersion effectiveness and energy savings for a new construction or a
cooling on information technology equipment reliability. The retrofit from device to the facility level. For mission critical
paper presents useful information regarding the influence of
operations of a data center, a comprehensive study of
mineral oil on the mechanical properties as well as chemical
properties of a material. The changes in properties of mineral oil reliability and availability is necessary for widespread
like kinematic viscosity and dielectric strength are also cited as adoption of this disruptive technology.
an important factor and discussed briefly. The changes in
mechanical properties like elasticity, hardness, swelling, and This study focuses on the reliability of servers and IT
creep are being shown in the paper for thermoplastic materials. equipment when submerged in a mineral oil at the device
The chemical reaction between material and mineral oil as a level. Prolonged immersion of servers in mineral oil will onset
function of time and temperature is also conferred. These are a wear-out mechanism and upon cumulative damage can lead
significant factors which are responsible for the reliability and to component failure. Degradation of material property or
compatibility of a material. The changes or modifications in component functionality is a result of fundamental
materials of important parts in physical and chemical manner
mechanical, chemical, electrical and thermal phenomena
are also indicated. The relations between different dielectric oils
and different materials provide us a comparative analysis for introduced due to changes in the typical operating
reliability and performance. environment. The superior heat carrying capacity of a mineral
oil compared to air eliminates hot spots and produces less
Oil immersion cooling of data centers offers opportunities for variation in temperature spatially and in time. The chemical
enhanced reliability with even temperature conditions in interactions between the coolant and various components for
operations as it minimizes common operational issues like: an extended amount of time introduce lifecycle loads not
overheating and temperature swings in the system, fan failures in observed in traditional air cooling. The components
servers, noise, dust, air quality, corrosion, electrochemical considered in this investigation are cables, printed circuit
migration, and whiskers will also be addressed. The reliability
boards, packages and passive components. When we deal with
improvements are comprised of the reduction in corrosion &
electrochemical migration like corrosive exposure and moisture immersion cooling it attacks to the reliability at device and
reduction, reduction in environmental contamination like dust, component level. With the concern of critical performance,
debris and particulate reduction, stable and even thermal cost, safety and operating environment, the study of the
environment and tin and zinc whisker mitigation. These may lead reliability of these four categories of components becomes
to infer the significance of oil cooled data centers towards significant. The study of the change in the properties of a
performance and savings in operating cost as per reliability mineral oil like kinematic viscosity, flash point, and dielectric
aspect. The literature gathered on the subject and quantifiable strength is also the subject of anxiety for the data center
data gathered by the authors provide the primary basis for this operators. These properties have the direct relation with the
research document.
coolant efficiency, servicing costs, pumping power, operating
cost and facility design. It becomes critical to know about
these changes as it keeps data centers functioning. Some
standard methodologies have already been derived from
measuring the change in properties of important components
submerged in a mineral oil. But sometimes, it becomes hard to
follow those standards because of following limitations [19.]:

1. The Joint Electron Device Engineering Council

(JEDEC) and the American Society of the
International Association for Testing and Materials
(ASTM) standards won’t be relevant due to the
significant difference in the ramp rates of air and oil.

2. Failure Mechanisms and Models for IT equipment

established by these standards are not directly
applicable in the reliability analysis of oil immersed
components as oil immersion cooling has different
operating conditions than the air cooled data centers
for which these standards have been derived.

3. IEC and ISO test methods might not have the direct
applicability to determine the real degradation in the
properties of a mineral oil as the oil cooled data center Figure 1: Plot for Temperature Vs Time for Air and Oil [20]
has different parameters affecting the operation such
as temperature, flow rate, varied surfaces, different Figure 1 shows the results of the trial test which was
materials etc. conducted to determine the feasibility of performing ATC
tests on oil in the environmental chamber. Clearly, the ramp
4. The air cooled system has a high fluctuation in time and dwell times between air and oil are going to be
operating conditions such as temperature and relative significantly different. Since it seems unlikely that accelerated
humidity. The standard also indicates such variations cycling will be able to be tested in a timely manner, an
in thermal cycling with high temperature differences. alternative test is sought. Elevated temperature tests (Thermal
In oil cooling, the temperature profile is more stable Aging) can be used to gather some results of plastic materials
and even. So the standard also needs to be developed for PVC, PCBs, and passive components. In this proposed
for such conditions and parameters should be modified method, we need to maintain a temperature above typical
accordingly. operating temperatures for an extended duration. This type of
test is common in capacitor degradation tests.
This paper also leaves a scope for instituting design of
experiments for determination of modeling parameters and a Oil immersion cooling technology of data centers extends
methodology which should be analogous to accelerated the prospects for improved reliability in operations as it
thermal cycling and accelerated thermal aging, so that it can minimizes common operational issues and eliminates the root
be accepted as a standard methodology to provide the causes of failure like reduction in solder joint failures, lower
reliability analysis of oil cooled data center components and operating temperatures for board & components, no
the coolant. The methodology should be proposed and adopted oxidation/corrosion of electrical contacts, no moving parts,
which can provide the reliable data to determine the failure in like fans, within the device enclosure, no exposure to
oil cooling technology. The assumptions should be made for electrostatic discharge (ESD), no sensitivity to ambient
all the parameters which are important in the case of oil particulate, humidity, or temperature conditions. The
immersion cooling. The parameters like heat load, flow rate, reliability advances include a reduction in corrosion &
inlet temperature, placement and power levels of the electrochemical migration, lessening of environmental
components and volume of the oil should be considered in contamination like dust, debris, and particulates, and
order to fix the temperature for the thermal overstress mitigation of tin and zinc whisker [5], [12] and [13].
experiment.

2
 II. EFFECT OF A MINERAL OIL ON PRINTED
CIRCUIT BOARDS AND PACKAGES

A primary concern by data center industry professionals

regarding mineral oil immersion techniques is the impact of
the fluids on the long-term reliability of components and
systems. By fully immersing a server in oil, a company may
be voiding the warranty on their equipment, and expose
themselves to potential failure costs. Current industry data
regarding the reliability of server systems after immersion in
mineral oil suggest that there is no detrimental impact to Figure 3: Comparison of microstructure of solder balls taken
components [6] [8]. However, the remarks made in literature from (a) an air cooled server and (b) an oil immersed server
are anecdotal, not providing detailed information or data, [21]
limiting their utility to the industry at large. Figure 3 offers a comparison of solder balls from the
backside of the memory module attached to the DIMMs. As
The study was undertaken and discussed below presents a can be seen, there are no noticeable deformations, change in
look at the impact of mineral oil on server components. This size, or cracking of solder balls. In addition, the intermetallic
includes high-level visual observations, microscopic compound (IMC) layers which provide the mechanical and
observations made by sectioning server components, and a electrical connection between PCB-solder balls and solder
more detailed study of the change in material properties that ball-substrate interfaces showed no change in thickness
result from exposure of printed circuit boards (PCBs) to between air cooled and oil cooled samples. The chip under fill
mineral oil. Similar observations of air cooled servers were material, which strengthens the mechanical connection
taken as a basis for comparison. between a flip chip package and substrate, also showed no
detectable variation between air and oil cooled samples. In
A sample of three servers that were immersed in mineral oil Figure 4, it is seen that there are no size variations in the metal
for a six-month period during thermal testing were taken apart, layers of the packaging a substrate. The trace thickness does
photographed and sectioned for imaging to document the not change or alter after exposure of the server in an
effects of oil on server components. Figure 2 shows the fading immersive environment.
off screen printed component markings on the memory chips
of the DIMM modules. Although not a direct impact on
mechanical reliability, this fading of markings may impact
identification of components, servicing should be needed.

Figure 4: Comparison of substrate layer of BGA package

taken from (a) an air cooled server and (b) an oil immersed
server [21]

Figure 2: Fading component identifies as a result of oil

exposure was seen in (a) an air-cooled server and (b) an oil
immersed server [21]

A more detailed visual study was carried out by taking

cross sections of various components to determine the
microstructure of electronic packages. Key components were
placed in molding compound, sectioned, and polished. Control
samples of servers that were not exposed to oil and used in
traditional air-cooled based testing underwent the same
testing. The details of the package structure were observed
under microscopes. Figure 5: Cross section of PCB plated through-hole on oil
exposed server [21]

3
 Samples of motherboard PCBs from air cooled and oil
immersed servers were taken and prepared for mechanical
testing using the Instron micro-tester. Preliminary stain
measurements showed a significant increase in the Young’s
modulus of PCB material from 27.2GPa to 38.1GPa for
servers that had been immersed in oil for an eight month
period. An increase of this type may severely limit the reliable
life a motherboard based on the trend discussed by Cheng et.
al. [3] and shown in Figure 7. The results from this study may
be input to Finite Element Models (FEA) to further simulate
the impact of changes in material properties on the component
and solder ball fatigue life.
Figure 6: Edge of oil exposed PCBs maintain structural
integrity and show no indication of delaminating [21] PCB dielectric constant and dissipation factor:
The properties those define any dielectric material of PCB
Additional samples of PCB boards showed no epoxies that have an effect on high-speed circuit design, in
delaminating, swelling, or warpage of layers after extended terms of signal propagation delay and dissipation factor are
periods of submersion in mineral oil. In Figure 5, a cross • Dielectric constant and
section at a plated through hole location on the motherboard • Dissipation factor or loss tangent.
has maintained its structural integrity. Similar observations The propagation delay in ns/m is given in terms of
can be made from Figure 6 at an edge location on the PCB. effective dielectric constant by the relation:
(1)
The images and results gathered here provide a more
detailed account to support the anecdotal claims made in the
literature. In terms of component reliability, when submerging where, is the effective dielectric constant of the PCB
servers over the six-month duration, there is not any indicated epoxy. The increase in the dielectric constant due to water
reason for concern. However, typical servers operate in a data (available in the form of relative humidity, water and gas
center for longer durations, anywhere from three years up to contamination in a mineral oil) absorption increases
10 years. A larger sample size of components and materials propagation delay, that is, it slows down the speed at which
tested over extended periods or with the aid of accelerated the signal travels down the wire.
thermal testing can help strengthen the conclusions made here. Dissipation factor or loss tangent tan(δ) affects the
An observation made when handling servers after submersion insertion loss (attenuation)α, in terms of dB/inch, of the signal
in mineral oil for extended periods is that some materials as per the following relation:
become noticeably stiffer. This includes the primary PCBs, as (2)
well as, the plastic and insulating materials used for
connection cords for power, networking, and hard disk drives where, f = frequency (GHz)
(HDD). A concern amongst industry professionals is that this
hardening may lead to cracking of insulators, exposing wiring As the frequencies rise up the GHz scale, more
or full failure of connectors. A study was initiated to consideration has to be paid to dielectric constant and the
determine the extent to which oil exposure alters the material dissipation factor of the PCB epoxies. The PCB epoxies
properties of PCBs. absorb moisture and will lead towards in an increment in the
values of both the dielectric constant and the dissipation
factor, degrading signal speed and increasing the insertion
loss. Fortunately, the moisture take up by the PCB epoxies is
slow and there are ample copper planes in PCBs to further
slow the moisture uptake. As well as, oil cooling technology
provides more protection to the PCBs with the direct exposure
to humidity in the atmosphere. Still, further studies are
warranted to study the effect of exposure to oil immersion
cooling on circuit performance.

Figure 7: Typical relationship between circuit board stiffness

and cycles-to-failure [3]

4
 III. EFFECT OF A MINERAL OIL ON CABLES AND resistant to heat and dielectric strength based on the test results
SWITCHING DEVICES provided by different esteemed laboratories.
The properties of insulation materials can be
characterized as extruded primary insulations, which are Properti PE
PVC XLPE mPPE Rubber
es Cable
applied directly over the conductor and also are used for
jackets. Those are mainly polyvinyl chlorides (PVCs),
polyethylenes, fluorocarbons, and silicones. While PVC Oil
insulation has desired mechanical and electrical properties and Resistan Fair Excellent Excellent Excellent Poor
ce
low cost, those make it a stronghold of the wire and cable
industry, it presents environmental concerns. PVC contains
Heat
halogen. PVC releases toxic gasses, smoke, and acids while Good-
Resistan Good Poor Good Fair
burning that can be harmful to health and equipment. XLPE is Excellent
ce
halogen-free but is not highly recyclable. These two materials
are abundantly being used in the cable industry. Using newly Dielectri
developed polyphenylene ether [mPPE] alloy insulating c Not Not
15-20 20 20
Strength Available Available
material is halogen-free and recyclable, yet remains cost- (kV/mm)
effective and robust. [9]
Table 1: Rating chart
For a Mineral oil immersed rack or tank, the cabling
architecture could comprise of a top of rack or end of row
design like all modern data centers [7]. The servers are
connected to the switch generally by an unshielded twisted
pair (UTP) cable for up to 10 GB/s for short distances. Off the
shelf Category 5E, 24 AWG UTP local area network cable
was considered for testing the impact of mineral oil. The cable
is plenum rated and has a low smoke PVC jacket and FEP
Figure 8: Electrical Cables and Wiring insulation. As the thickness of the jacket specimen was less
than 0.76mm, tubular specimen was prepared and tested in
accordance with UL 2556. The specimens were 6 cm long and
immersed in mineral oil for 48 hours at 100°C. The
mechanical testing of the cable jacket specimen (the test setup
has been shown in Figure 10) was performed to determine the
percentage change in elongation and to compare the change in
Young’s Modulus.

Figure 9: Network Cables

There are different formulations of Polyvinylchloride

which show extremely high or low-temperature properties of
PVC. Some PVC formulations have –55°C to 105°C rating.
The regular PVCs have –20°C to 60°C. The dielectric constant
is in between 3.5 to 6.5. XLPE is having 150°C ratings. Cross-
linking converts polyethylene to a thermosetting material
which enhances the properties of a material.

Rubber can be categorized as natural rubber and SBR

compounds. These materials can be used for both insulations
and jackets purposes. Some formulations are suitable for –
55°C minimum, while others are suitable for 75°C maximum. Figure 10: Test setup to measure mechanical properties of
cable specimen
Table 1 provides information regarding the properties of
general insulation and jacket materials like oil resistance,

5
 From Figure 11, it can be inferred that there is a drastic from the body of the capacitor thereby increasing the internal
increase in Young's Modulus. Due to aging in mineral oil the temperature.
specimen shrunk in length and loss of plasticizers could be
attributed to its reduction in weight.

Figure 13: Capacitor degradation data in a thermal overstress

experiment from air cooling testing [16]

Figure 11: Load versus extension for a Low smoke PVC Figure 13 describes capacitor degradation data in a thermal
jacket tubular specimen without aging and aged in mineral oil overstress experiment at 105°C and humidity factor of 3.4% for
3500 hours shown below. Electrolytic capacitors and Polymer
capacitors are mainly used in the data center industry and
should be tested in Mineral oil at elevated temperatures as
provided for air cooling testing in figure 13 [16].

Degradation of Capacitors in turn leads to an implication in

two main electrical parameters of the capacitor:
1. Equivalent Series Resistance (ESR), and
2. Capacitance (C).

The temperature profile of a server, when immersed in

mineral oil, reduces hot spots and ΔT across the servers and,
therefore, provides a better operating environment for
capacitors. The concern in mineral oil is about the dissolution
Figure 12: Ethernet switches immersed in oil [13] of the electrolyte in mineral oil causing degradation in
No performance impact of a mineral oil on RF performance. Rubber bungs at the bottom and plastic capacitor
components and switching devices has been found yet. No sleeve should be avoided due to incompatibility with mineral
considerable analysis has been carried out yet. Operators have oil. A degradation of capacitance should indicate any decrease
not detected any issues, but to predict the life of materials and in electrolyte volume of the capacitor. Liquid electrolyte
performance, the study should be acted upon. capacitors that produce hydrogen gas when it fails due to the
chemical reaction inside can be a cause of concern as well.

IV. EFFECT OF A MINERAL OIL ON PASSIVE V. CHANGES IN PROPERTIES OF A MINERAL OIL

COMPONENTS LIKE CAPACITORS The standard properties of mineral oil as per Data
Electrolytic capacitors, prominently the Aluminum sheets/MSDS from STE Oil Company Crystal Plus Tech Grade
Conductive polymer capacitor, are generally used in servers. Mineral Oil are:
Degradation of performance in electrolytic capacitors can be Density (kg/m3) : 849.3
caused by various factors like electrical, thermal and Specific Heat (kJ/kg∙K) : 1.670
environmental stresses. Upon electrical overstress, the increase Thermal Conductivity (W/m∙K) : 0.13
in internal temperature in turn increases the electrolyte Thermal Expansions Coefficient (1/K) : 0.0007
evaporation rate. Similarly, when the capacitor is operating or Thermal Diffusivity (m2/s) : 9.166E-08
is stored in the high-temperature environment the heat travels Prandtl Number (at 40°C) : 134.4

6
 NOTE: Some of these properties may be temperature
dependent (i.e. density). The changes in properties of a mineral
oil are mainly concerned with the changes in kinematic
viscosity and dielectric strength.
The viscosity of oils has a relationship with temperature
and time and it affects system pressure directly. So that
pumping power becomes critical. A standard correlation
between viscosity and temperature for transformer oils is given
in [10] as:
(3)

where, µ = dynamic viscosity (centipoise),

T = temperature (°C), and C = coefficient for scaling

Oil Dynamic Kinematic

Temperature Viscosity Viscosity Figure 14: The relationship between pumping power and
(°C) μ (kg/m·s) ν (m2/s) temperature dependent dynamic viscosity [10]
Since pumping power and flow rate are directly related to
30 0.01405 1.65E-05 the operating cost, critical performance and efficiency of
facility equipment, to study the phenomena regarding the
change in viscosity of a mineral oil for data center operators
becomes significant. Temperature, oxygen availability, and
35 0.01209 1.42E-05 presence of a catalyst (Thermal Aging) are the main factors
that influence the chemical stability of oil. The hydrocarbon
molecules in oil start decomposition at high temperature and
that may cause the oil degradation process. The oxygen
40 0.01046 1.23E-05 contents in cooling oil might lead to a rise of the acidity
number and to sludge formation. Catalysts such as copper and
iron are dissolved in oil during aging and might accelerate the
45 0.00909 1.07E-05 aging process [17].
During an accelerated thermal aging process, It is also
important to analyze the dielectric properties of mineral oil,
50 0.00794 9.35E-06 such as the breakdown voltage, dielectric losses (tan A), and
relative permittivity. There is some analysis carried out for
55 0.00696 8.19E-06 transformer oil. The study of aging test and dielectric property
analysis for the transformer (mineral) oil concludes that there is
Table 2: Analytical calculation of Viscosity with respect to a chance of a leak during the operation and its less
Oil Temperature biodegradable property leads towards pollution. It has a low
The direct proportionality of viscosity with Reynolds flash point and high pour point for transformer operations. At
Number (Re), Reynolds number with friction factor (f), and the end of the aging process, the mineral oil demonstrated
friction factor with pressure drop (Δp) for laminar flow is given lower breakdown voltages rather than at the starting. The tan δ
of the mineral oil showed the significant variation during the
below as [10],
different stages of aging test process. The effect of humidity
(4) should also be considered during the oil aging test [17].

That results in the relation with the pumping power, Figure 15 indicates the dielectric strength of transformer oil
(mineral oil) remains fairly consistent across the temperature
(5) range of interest for data center applications and Figure 16
shows the study of change in the dielectric property of oil in
where, = volumetric flow rate [10]. transformers which shows some degradation over time. The
Figure 14 infers that the change in viscosity is having the literature states breakdown voltage depends on water content,
direct impact on pumping power that may be useful to derive suspended particles, and cleanliness.
the flow rate and operating conditions. It has a direct relation
with operating cost too.

7
 Figure 17 provides the glance of the working of oil cooled
data centers, which eliminates the root causes of failures,
improves the operating conditions and reliability and advances
in cooling technology for data center industries. This
technology paves the path for the retrofitting of air cooled data
centers and efficient performance with high load capacities.
Figure 17 is an experimental setup that gives the understanding
of the working environment of oil cooled data centers that may
encourage its adoption by operators with its simplicity in
implementation.

Figure 15: Temperature Dependence [11]

Figure 16: Breakdown over Time [11]

Thus, the dielectric property analysis for data center Figure 17: Experimental Setup providing the glimpse of oil
cooling mineral oil should be carried out by proposing an aging cooled data center [21]
testing methodology to provide the data for variation in
properties to operators for advancements in oil cooling The reliability upgrading of data center operations is mainly
technology and to increase its applicability. It would be an [12], [18]:
interesting study to observe the change in kinematic viscosity 1. Lessening the Electrochemical Migration
and thermal conductivity of a mineral oil during the actual
operation as a function of temperature and time. The • Reduction in corrosive exposure
comparative study of different engineered fluids should also be • Reduction in moisture
performed to provide more options to industries.
2. Reduction in Contaminants
VI. POTENTIAL RELIABILITY ENHANCEMENTS • Efficient handling and cleanliness prevent
Oil cooling technology offers minimization in common accumulation of particulates and dust
operational issues and removal in failure modes like [18], [20]: 3. Temperature stability
•Overheating 4. Mitigation of Tin and Zink Whisker
•Temperature swings Electrochemical Migration is a movement of metal through
• Failures of server fan an electrolytic solution under an applied electric field between
conductors which are insulated. Electrochemical migration
•Solder joint failures
(ECM) is a common reliability issue that can be found in the
•Air quality electronic packaging industry, including different materials and
components like dies surface, epoxy encapsulates, PCB and
•Dust
passive components etc. The major ECM drivers are
•Corrosion temperature, moisture, contamination and voltage/electrical
field. Immersion oil cooling reduces and/or eliminates
•Whiskers
temperature, moisture, and contamination aspects [14], [13].

8
 The main sources of contamination are storage and “Unfortunately, accelerated techniques do not currently exist
environment. The concerns of storage include cleaning to predict if, when, and to what extent a zinc-coated surface
chemicals, outgassing and polymeric materials. Operating will produce zinc whiskers.”[15]
environments include dust, debris, zinc whiskers, moisture, “If you manage a data center, especially one that sits on a
evaporated sea water and industrial pollutants like sulfur, etc. raised floor, zinc whiskers might eventually have an impact on
Oil immersion cooling prevents contamination accumulating. your operations.”[15]
Efficient handing methods and cleanliness should be
implemented and filtration of oil lessens the risk of particulate
and dust contaminants [14].

Figure 20: Zink Whiskers [15]

For Zink whiskers, there is some uncertainty persist

Figure 18: Thermal profile of operation for oil cooled data regarding the prediction of whiskers and its criticality of
center failure modes [15]. But still, the study may provide some
Figure 18 shows the thermal profile for a temperature cycle noteworthy information regarding its impact on data center
with 24-hour server uptime. The results show the constant operations.
operation that proves the thermal stability of oil cooled data
centers. It is expected to see a hot spot reduction and improved VII. CONCLUSION AND FUTURE WORK
thermal uniformity using immersion oil cooling [12] [13]. The information furnished here is based on strong
literature review and quantifiable data gathered for validation.
The study provides a strong background for direct and indirect
reliability concerns related to oil cooling technology. The
operators will have trustworthy data to implement this
technology.

Oil immersion cooling may offer better practices in some

developing countries like India, China, etc where the
environmental conditions for data centers are above ASHRAE
Figure 19: Tin whiskers G3 severity level and where it is hard to implement airside
economizers to derive recommended environmental envelope.
Figure 19 describes the phenomenon of Tin whiskers, With the enhanced reliability, high heat dissipation and
hair-like single crystal metallic filaments that grow from Tin performance efficiency, oil cooling technology may serve the
films. The potential failure modes are direct contact that data center cooling technology world as a leader in the future.
causes an electrical short (arching) and requires growth of
adequate length and in the correct direction. Electro-magnetic As we can reiterate that the field of reliability for oil
(EM) Radiation which releases or receives EM signal and immersion cooling is having a lot of scope for future work.
noise at higher frequencies and deterioration of signal for The effect of mineral oil on major components should be
frequencies above 6 GHz, which is independent of whisker measured. The changes in mechanical and chemical properties
length and debris like whisker breaks off and shorts two leads of mineral oil should be investigated by aging test. The
(primarily during handling). These can be mitigated by humidity and contaminant barriers, especially leaching out
immersion oil [12], [13]. from plastics of cables and components should be checked.
The temperature and heat density optimization should be
The occurrence of Zink whiskers is shown in figure 20. carried out. Thus, the scope of studying thermal performance

9
and reliability concerns is of all data center operators' interest 18. J. Shah, et. al., “Critical non-thermal consideration for oil
and concern. cooled data-center”, IMAPS ATW 2015, Los Gatos, Ca, 2015.
REFERENCES 19. Jimil M. Shah and Dereje Agonafer, “Issue on Operational
Efficiency for Oil Immersion Cooled Data Centers”, Session
1. Green Revolution Cooling (http://www.grcooling.com/)
Co- Chair and Presenter for ASME Panel On "Thermal
2. Midas Green Tech (http://www.midasgreentech.com/)
Management Challenges in Energy Conversion &
3. http://www.liquidcoolsolutions.com/
Conservation", ASME IMECE 2015, November 13-18,
4. http://green.blogs.nytimes.com/2012/09/06/cooling-a
Houston, Texas.
computer-server-with-mineral-oil/
20. Jimil M. Shah, “Reliability challenges in airside
5. D. Prucnal, "Doing More With Less: Cooling Computers
economization and oil immersion cooling”, The University of
with Oil Pays Off," The Next Wave, vol. 20, no. 2, pp. 20 - 29,
Texas at Arlington, May 2016.
2013.
21. Richard Eiland, “Thermo-Mechanical Design
6. Submersion Cooling Evaluation, PG&E’s Emerging
Considerations at the Server and Rack Level to Achieve
Technologies Program;
Maximum Data Center Energy Efficiency”, The University of
http://www.etccca.com/sites/default/files/reports/E13PGE110
Texas at Arlington, May 2015.
1%20Submersion%20Cooling%20for%20Data%20Centers_1.
pdf
7. http://www.thefoa.org/tech/ref/appln/datacenters.html
8. H. Cheng, et. al. “ Parametric Analysis of Thermally
Enhanced BGA Reliability Using Finite Volume Weighted
Averaging Technique” ASME IMECE, 1998
9. http://www.ecswire.com/tools/insulation_jackets/characte
ristics/#CSPE
10. R. Eiland, J. Fernandes, M. Vallejo, D. Agonafer and V.
Mulay, "Flow Rate and Inlet Temperature Considerations for
Direct Immersion of a Single Server in Mineral Oil," in IEEE
ITHERM, Lake Buena Vista, FL, 2014.
11. Clark, F. M., “Dielectric Strength of Mineral Oils”
Electrical Engineering, Volume: 54, Issue: 1, 1935, pages 50 -
55, DOI: 10.1109/EE 1935.6539592
12. C. Tulkoff and C. Boyd, "Improved Efficiency &
Reliability for Data Center Servers Using Immersion Oil
Cooling," in Electronic System Technology Conference &
Exhibition, Las Vegas, NV, 2013.
13. www.dfrsolutions.com
14. Bergles, A.E., and Bar-Cohen, A., Immersion Cooling of
Digital Computers, Cooling of Electronic Systems, Kakac, S.,
Yuncu, H., and Hijikata, K., eds, Kluwer Academic
Publishers, Boston, MA, pp. 539-621, 1994.
15. “Zinc Whiskers: Could Zinc Whiskers Be Impacting Your
Electronics?” J. Brusse, Apr. 2003; http://
nepp.nasa.gov/whisker/reference/tech_papers/Brusse2003-
Zinc-Whisker-Awareness.pdf
16. Kulkarni, et. al., “Physics Based Electrolytic Capacitor
Degradation Models for Prognostic Studies Under Thermal
Overstress”, European Conference of the Prognostics and
Health Management Society, 2012.
17. Endah Yuliastuti, "Analysis of dielectric properties
comparison between mineral oil and synthetic ester oil," Delft
University of Technology, June 2010.

10

View publication stats