Acta Materialia 168 (2019) 473e510

Contents lists available at ScienceDirect

Acta Materialia
journal homepage: www.elsevier.com/locate/actamat

Overview article

Materials informatics: From the atomic-level to the continuum

J.M. Rickman a, b, *, T. Lookman c, S.V. Kalinin d, e
a
Dept. of Materials Science and Engineering, Lehigh University, Bethlehem, PA, 18015, USA
b
Department of Physics, Lehigh University, Bethlehem, PA, 18015, USA
c
Theoretical Division, Los Alamos National Laboratory, Los Alamos, NM, 87545, USA
d
Center for Nanophase Materials Science, Oak Ridge National Laboratory, Oak Ridge, TN, 37831, USA
e
Institute for Functional Imaging of Materials, Oak Ridge National Laboratory, Oak Ridge, TN, 37831, USA

a r t i c l e i n f o a b s t r a c t

Article history: In recent years materials informatics, which is the application of data science to problems in materials
Received 22 October 2018 science and engineering, has emerged as a powerful tool for materials discovery and design. This rela-
Received in revised form tively new field is already having a significant impact on the interpretation of data for a variety of
28 January 2019
materials systems, including those used in thermoelectrics, ferroelectrics, battery anodes and cathodes,
Accepted 28 January 2019
hydrogen storage materials, polymer dielectrics, etc. Its practitioners employ the methods of multivariate
Available online 30 January 2019
statistics and machine learning in conjunction with standard computational tools (e.g., density-
functional theory) to, for example, visualize and dimensionally reduce large data sets, identify pat-
Keywords:
Data analytics
terns in hyperspectral data, parse microstructural images of polycrystals, characterize vortex structures
Microstructure in ferroelectrics, design batteries and, in general, establish correlations to extract important physics and
Ferroelectrics infer structure-property-processing relationships. In this Overview, we critically examine the role of
Electron microscopy informatics in several important materials subfields, highlighting significant contributions to date and
Battery materials identifying known shortcomings. We specifically focus attention on the difference between the correl-
ative approach of classical data science and the causative approach of physical sciences. From this
perspective, we also outline some potential opportunities and challenges for informatics in the materials
realm in this era of big data.
© 2019 Acta Materialia Inc. Published by Elsevier Ltd. All rights reserved.

Contents

1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 474
2. Adaptive design for materials discovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 475
2.1. Historical perspective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 475
2.2. The materials information landscape . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 478
2.3. High-throughput screening via ab initio calculations and databases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 478
2.3.1. Search for thermoelectric and battery materials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 479
2.3.2. Stability in high-throughput computations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 480
2.3.3. Search for perovskites and glasses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 480
2.4. Inverse design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 482
2.5. Adaptive learning for materials discovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 482
2.5.1. Bayesian Global Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 483
2.5.2. Utility functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 484
2.5.3. Applications to materials problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 485
3. Data analytics and image processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 488
3.1. Order-parameter mapping and local crystallography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 488
3.2. Analysis of multispectral data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 489

* Corresponding author. Dept. of Materials Science and Engineering, Lehigh

University, Bethlehem, PA, 18015, USA.
E-mail address: [email protected] (J.M. Rickman).

https://doi.org/10.1016/j.actamat.2019.01.051
1359-6454/© 2019 Acta Materialia Inc. Published by Elsevier Ltd. All rights reserved.
474 J.M. Rickman et al. / Acta Materialia 168 (2019) 473e510

3.3. Functional recognition imaging and physics extraction using neural networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 492
3.4. Deep learning for defect and chemical identification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 494
3.5. Physics extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 495
3.5.1. Reaction-diffusion phenomena . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 495
3.5.2. Physics of ferroelectrics and vacancy-ordered systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 496
3.5.3. Discrete systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 497
4. Future opportunities in atomically-resolved imaging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 498
5. Microstructural informatics: classification and characterization of materials microstructures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 499
5.1. Tools for microstructural quantification and reconstruction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 500
5.2. Microstructural classification and property linkage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 501
5.3. Quantification and analysis of abnormal grain growth . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 501
5.3.1. Example: processing of specialty alumina powders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 502
6. Discussion and outlook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 503
Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 504
Appendix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 504
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 505

1. Introduction suitability of materials for battery applications, other workers have

enumerated thermodynamically stable lithiation reactions of
We are now inundated with massive amounts of data resulting transition metal silicides, stannides and phosphides, computing the
from emerging computer technologies, scientific tools, the Internet cell potential, capacity and volume expansion for each of these
of things, etc., and it can be expected that this data avalanche will reactions [12]. Materials informatics methods have also been used
continue unabated [1]. Big data is the term that captures this large in the search for lead-free inorganic/organic hybrids in layered,
volume of both structured and unstructured information, and its double perovskites, and metal-deficient forms have recently been
utilization is having an impact on science and engineering due to reviewed by Chakraborty et al. [13].
the promise of being able to tackle complex problems [2] including, However, several challenges confront workers who wish to
for example, the monitoring of movable bridges [3], semiconductor apply informatics methods to their problems of interest. First, many
manufacturing [4] and the production of flat steel [5]. To harness of the methodologies have been developed primarily outside the
and utilize this explosively growing volume of data, data scientists field of materials science and typically require either domain
employ statistics and the techniques of computer science, such as expertise in statistics and computer science or a substantial, up-
artificial intelligence, to identify patterns and gain insights that front time investment to learn what is required. To complicate
permit better decision making [6]. Thus, in recent years, data sci- the task, the new lexicon associated with these techniques is also
ence has emerged as the “fourth paradigm” [7] in scientific replete with jargon and acronyms. Second, potential users must
research, joining experiment, theory and computation to comprise decide whether employing informatics techniques will yield any
a fundamental framework for inquiry. new insights in a particular context. In short, one must determine
The creation of large data sets in materials science, in particular, what new physics may be learned in a given application and how
is transforming the way research is done in the field by providing this physics (i.e., causative and quantitative relationships) can be
opportunities to identify complex relationships and to extract in- inferred from correlative analyses. Finally, there are issues associ-
formation that will enable new discoveries and catalyze materials ated with the management of datasets, sometimes acquired with
design. More specifically, the application of data science to mate- multiple platforms, and the visualization and interpretation of this
rials problems has been termed “materials informatics”, an data. These datasets are often large and may sometimes be ob-
approach to materials discovery that allows a high-throughput tained in a relatively short period of time.
analysis of multiscale information [8]. In recent years, the tools of It is worth noting, however, that, in some cases, materials sci-
data science, which include multivariate statistical analysis, ma- entists are confronted with relatively small, heterogeneous data
chine learning, dimensional reduction and data visualization, have, sets, and the aim is to extract maximum information from limited
for example, led to 1.) the identification of important structure- data [14]. In particular, existing data is often confined to specific
property-processing relationships [9], 2.) the screening of prom- regions of parameter (e.g., composition) space or, in the case of
ising alloys for various technical applications [10] and 3.) the cor- computational data, may be available only for a narrow range of
relation of microstructure with processing parameters. Given the systems [8]. Moreover, these small dimensional data sets often
increasing importance of materials informatics in our field, we result from experimental exploration of a large dimensional
believe that a broad review of the impact, successes, challenges and parameter space with only the final result available in the form of
opportunities associated with data-intensive science in the mate- data and publications. Nevertheless, in recent years, researchers
rials community is timely. have sought to use materials informatics to gain insights from small
As indicated above, this nascent field is already having a sig- data sets. For example, Balachandran et al. [15] searched for new
nificant impact on materials discovery and data interpretation in a functional materials using a data analytics-guided ab initio
variety of materials systems including those used in thermoelec- approach to materials discovery. Their focus was on promising
trics, ferroelectrics, battery anodes and cathodes, hydrogen storage Ruddlesden-Popper oxides, of which only a small number have
materials, polymer dielectrics, etc. For example, data mining has desirable properties. We note, however, that the application of
enabled workers to extract information from a large database of analytical techniques, such as machine learning, to small data sets
calculated properties for half-Heusler alloys to determine, via a presents certain risks, including the danger of over-fitting and
figure-of-merit calculation, whether a given alloy has the potential concomitant losses in predictive capabilities [16,17]. Workers have,
to be a good thermoelectric [11]. In addition, to assess the in fact, just begun to assess the importance of data set size on the
 J.M. Rickman et al. / Acta Materialia 168 (2019) 473e510 475

accuracy of predictions associated with different learning strate- of identifying new semiconductors. In this regard, a quote from
gies. In this regard, Schmidt et al. [18] examined the role of training Chelikowsky and Phillips [21], who studied the classic problem of
set size on calculations of the thermodynamic stability of solids that classifying the AB sp-bonded octet solids in the 1970s, is particu-
combined density-functional theory (DFT) and machine learning. larly apt: “Structural energy differences are, for the most part, too
They found that about 20,000e30,000 training samples are small to be calculated quantum mechanically, because such calcu-
required to accurately determine the energy per atom, and that the lations require self-consistent crystal potentials of an accuracy
predicted error decreased slowly with the size of the larger training beyond the present state of the art. However, if we consider the
sets. problem from the point of view of information theory, then the
In this Overview we will examine the role of informatics in available structural data already contain a great deal of informa-
several important materials subfields in which it is already having a tion: about 120 bits, in the case of the AB octet compounds. Thus,
substantial impact, including: the use of Bayesian methods to one can reverse the problem, and attempt to extract from the
inform manufacturing, the use of analytics on multispectral images available data quantitative rules for chemical bonding in solids.”
and data mining in multimodal imaging, the application of density- Clearly, these authors recognized that the energy differences be-
functional theory and machine learning to high-throughput alloy tween structures obtained using nonlocal pseudo-potentials were
design, the tailoring of microstructures in nanostructured mate- often too small (~0.1% of the cohesive energy) to be calculated in
rials, the characterization of microstructures and their influence on that era, and they suggested instead that examining the problem
thermomechanical properties and the creation of improved atom- from an informatics point of view to learn rules from bonding data
istic potentials for materials simulation. We specifically focus would be fruitful. They had a database of some 80 compounds and,
attention on the difference between the correlative approach of from this information, proceeded to construct 2D structural maps.
classical data science and the causative approach of the physical Somewhat earlier, Philips and collaborators [19,22,23] had already
sciences and, for this purpose, present several examples of physics been thinking about descriptors for AB compounds, focusing on
extraction from imaging data and the use of known physical priors establishing a minimal set of (in their case two) descriptors. These
to enable image analysis. Our overarching aim is to provide the Acta descriptors included those based on microscopic theory and
readership with a snapshot of the field that is significantly broader available spectroscopic data, such as the principal quantum num-
and more critical than existing reviews, along with current infor- ber of the two atoms, the Pauling electronegativity difference [19]
mation as to best practices and potential opportunities. To assist the of the atoms, the average covalent energy gap and the average ionic
reader in navigating this survey, we also supply an Appendix con- energy gap [22]. With such descriptors, Philips et al. were already
taining a short glossary of relevant terms and methodologies. able to achieve 90% success in separating four-fold and six-fold
This Overview is organized as follows. In Sec. II, we present an coordinated compounds.
outline of adaptive design for materials discovery, beginning with a Other groups also examined possible chemical descriptors. For
brief historical perspective, followed by a discussion of high- example, St. John and Bloch [20] proposed the use of the pair ( rs ,rp
throughput screening using ab initio calculations and databases ) as a linear combination of s and p
orbital-depended



radii, rs and

rp ,






and a summary of progress to date in the discovery of thermo- of A
and

B atoms
defined by rs ¼
r Ap þ r As

r Bp þ r Bs
and rp ¼
r Ap 
A

B B

electrics, battery materials and perovskites. In Sec. III, we review r s
þ
r p  r s
. This combination of radii has successfully provided a
recent contributions of data analytics to image processing. In basis for further work on the structural classification of AB solids for
particular, we begin with a short discussion of order-parameter a wide range of chemistries [21,24,25]. Fig. 1 from Chelikowsky and
mapping and local crystallography and then consider the data Phillips [21] shows the 2D St. John-Bloch plot in terms of the orbital
analytical approach for the exploration and analysis of hyper- radii of the symmetry-adapted combinations of s and p atomic
spectral data. Several other specialized topics in image processing orbitals calculated using nonlocal pseudopotentials. Although these
are also reviewed, including functional recognition imaging and radii are not directly obtained experimentally, they provide a
physics extraction using neural networks, deep learning as applied measure of bond valency and orbital electronegativity. These initial
to defect recognition and chemical identification and physics studies emphasized the construction of a structure (i.e., x  y) plot
extraction in various systems (e.g., ferroelectrics). In Sec. IV, we with the values of selected descriptors or physical features of the
consider the application of materials informatics to the classifica- materials placed along the x  y axes. The aim is to identify two
tion and characterization of materials microstructures. In this sec- features amongst a list of atomic radii, electronegativity, ionization
tion, the role of informatics in the quantification and reconstruction potentials, etc. for the AB solids that allow one to draw lines with a
of microstructures is highlighted, along with connections between pencil and a ruler so that one can cluster materials having the same
microstructure and physical properties. As an illustration of the crystal structure. Empty spaces in the resulting cluster plot then
power of this methodology, the challenge of quantifying abnormal suggest possible new materials and their associated crystal struc-
grain growth is examined in some detail. Finally, Sec. V contains a tures. In this regard, it would be remiss not to also mention Petti-
short discussion of the current state of materials informatics and an for's chemical scale [26], which led to the Mendeleev numbers, that
outlook emphasizing future challenges and opportunities. permitted a single 2D map to be plotted for all binary compounds
with a given stoichiometry. Each point in the plot is a pair of
2. Adaptive design for materials discovery numbers and, although quite empirical, this general scale separates
octet and non-octet AB solids structurally, and separates four-fold
2.1. Historical perspective and six-fold coordinated octet sp-bonded solids almost perfectly,
as can be seen from Fig. 2. The numbers provide an ordering to the
One of the first examples of the use of information sciences in elements in the periodic system, with the aim of arranging chem-
materials design occurred more than 50 years ago in the study of ical elements having similar behavior in succession (in contrast to
crystal structure classification of the octet subset of AB binary solids the atomic number ordering).
into rocksalt, zincblende, wurtzite, cesium chloride, and diamond Recently, several groups have revisited the problem of classifi-
structures, with rocksalt being the most common structure [19,20]. cation of the four-fold and six-fold coordinated octet AB
The octet AB compounds are defined by AN B8-N, where N is the spebonded compounds from a statistical learning perspective us-
number of valence electrons. Their study was motivated by a need ing machine learning tools. For example, Saad et al. [27] used the
to understand the nature of chemical bonding in solids and the goal orbital radii pairs in addition to a larger set of features. A different
476 J.M. Rickman et al. / Acta Materialia 168 (2019) 473e510

Fig. 1. The St. John-Bloch plot for the binary octet crystals ANB8-N that separates crystal structures using the radii ðrs ; rp Þ, defined in terms of the s and p orbital radii for the A and B
atoms (reproduced from Ref. [19]).

Fig. 2. (a) The chemical scale, c, or Mendeleev numbers provide an ordering to each chemical element in the periodic table with the aim of arranging them so that chemical
elements with similar behavior follow each other. (b) The map ð cA ; cB Þ for 574 binary compounds AB shows excellent structural separation amongst the 34 structural types
(reproduced from Ref. [26]).
 J.M. Rickman et al. / Acta Materialia 168 (2019) 473e510 477

strategy was employed by Ghiringhelli et al. [28] who used a In the materials engineering literature, the Ashby plot (Fig. 3)
combination of atomic features to construct a particular optimal [31] is a well-known example of a 2D scatter plot that is often
pair from the atomic ionization potential, electron affinity and radii. employed to select the “best” material for use in a particular
Pilania et al. [29] found in their work that the Born effective charge, application. In practice, two properties of many materials forming
a measure of the change in polarization due to displacement of a multiple materials classes are plotted on the same axes. (For
charge, was a very effective descriptor, in conjunction with rs as example, Ashby plotted the Young's modulus versus the density for
well as separately with the bond polarity [30], for accurately many metals, polymers, ceramics, foams, etc.) Given a specific
separating rocksalt and non-rocksalt structures. All of these recent application, a choice can then be made of the materials class and
studies employed standard machine learning algorithms, such as the specific material in that class that are most suitable in a given
support vector machines as well as kernel ridge regression and context. The characteristic features for each material class in an
gradient tree boosting. Unlike the work decades ago, these tools Ashby plot are limiting boundaries (e.g., high Young's modulus and
provided estimates of the average classification accuracy and the low density, high Young's modulus and high density, etc.). Thus, the
associated variances in the inference model where a decision search for new materials involves finding those candidates that
boundary separating the structures is learned in a supervised push one of these boundaries in a favorable direction. These
manner. The boundaries are no longer straight lines drawn by boundaries are called Pareto fronts, and the design approaches we
hand; instead, these methods are a generalization of the analysis of will review here are being generalized to find materials in which
the 2D structure plots to an arbitrary number of descriptors in a two properties or objectives must be optimized together.
multidimensional space. This is the significant advantage that Recently, the materials genome initiative (MGI) has catalyzed
modern computation and tools developed in the information sci- considerable work in the US and many parts of the world with
ences has brought to materials science. activities ranging from materials discovery to property

Fig. 3. Ashby plot of the loss coefficient versus Young's modulus for different classes of materials. The loss coefficient measures the internal friction or damping, an important
material property when structures vibrate or dissipate energy. In metals, a large part of the loss is hysteretic, caused by dislocation movement; it is high in soft metals such as lead
and aluminum. Alloyed metals like bronze and high-carbon steels have relatively low loss because solute atoms pin dislocations. Exceptionally high loss is found in the Mn-Cu alloys
because of a strain-induced martensitic transformation. In polymers, chain segments slide against each other when loaded; their relative motion dissipates energy. (reproduced
from Ref. [31]).
478 J.M. Rickman et al. / Acta Materialia 168 (2019) 473e510

optimization to system design and integration, all the way to established. Neural net and deep learning methods have an
deployment [32]. Although this whole pipeline is integral to the important role to play in accelerating the process of automatic
stated MGI objective of “twice as fast at a fraction of the cost”, the curation and extraction of data from raw instrument data flow, and
discovery and optimization aspects are crucial initial conditions, some of the advances in machine learning for computer vision and
leading to the obvious question: How do we accelerate the process imaging can be applied in this field [41].
of selecting chemistry, composition and processing conditions to The materials data landscape shown in Fig. 4 thus spans the
find materials with targeted properties? There are several chal- “big” data in the 10e100 TBs/sample range generated at major user
lenges here. First, the reason that materials discovery is even an facilities to the small-scale experiments performed at most labo-
issue is that the discovery search space is typically very high- ratories in the world, the latter involving standard bulk synthesis
dimensional. In short, there are millions of possible compounds and characterization on a relatively limited numbers of samples
in different structural classes (e.g. perovskites), each class con- (10e100s). Moreover, new facilities that are being planned, such as
taining many dopants or phases (solid solutions, binary, ternary LCLS II, the upgrade to LCLS, and the DOE/LANL signature concept
alloys) of which only a tiny fraction has been experimentally MaRIE (Matter Radiation Interaction in Extremes), for in situ spatial
explored [8]. This space includes many different chemistries, crystal and time-resolved measurements on polycrystals, will increase the
structures, processing environments and microstructures. Second, existing data output by a factor of 10 and 20 to PBs per sample as
the material descriptors or attributes associated with a particular compared to TBs at today's facilities. There are, however, some
sample or entity in a database can be dependent on features strategies for addressing this data avalanche. For example, the
spanning several length and time scales, from atomistic (e.g., output of calculations, such as that provided by materialsproject.
atomic radius, number of valence electrons per atom) to larger org, OQMD, AFLOWLIB, CALPHAD and finite-element and phase-
length scales (e.g., polarization, permittivity) and from slow to fast field codes, permit one to fill in some data gaps by augmenting
kinetics. Third, one wishes to optimize materials properties. In experimental materials data with complementary and supple-
general, optimization involves regression algorithms, and there are mentary computational data, typically in the GBs-TBs range. In
now suites of open source software, such as SCI-KIT LEARN and addition, imputing missing data to build inference models, while
TENSOR FLOW, readily available for application (subject to the se- not well studied [42,43], is another informatics strategy that may
lection of proper training sets and network configurations for a enable progress towards predictions for future experiments.
specific problem). However, uncertainties are intrinsic to measured Although future characterization facilities are expected to generate
estimates of the descriptors as well as associated properties, and massive amounts of data, for many materials problems the datasets
these uncertainties will clearly influence any models constructed will correspond to very localized regions in the materials space.
from the data [33]. Since data is the key ingredient in any learning
strategy, we next examine the scope and sources of data employed 2.3. High-throughput screening via ab initio calculations and
in materials science. databases

2.2. The materials information landscape A significant amount of effort has been expended over the last
few years to generate libraries of data using ab initio codes, such as
Many studies in the emerging area of materials informatics have VASP and QUANTUM ESPRESSO, on different classes of compounds
focused on establishing a database of crystal structures and asso- in a high-throughput manner. In this approach to design one ob-
ciated chemistries to define the space of the training data, including tains data on all, or a large number, of combinatorial possibilities
descriptors such as symmetries, atomic radii, bond angles and and then successively down-selects to focus on a relatively small
lengths and energetics. Data generated from computational codes, subset of promising materials for further study and/or for making
including band gaps and densities of states from electronic struc- experimental recommendations. Beyond the aforementioned
ture calculations, have been one of the main sources of information materialsproject.org, AFLOWLIB and OQMD, relatively newer re-
for database projects, such as materialsproject.org [34,35], OQMD positories, such as the Novel Materials Discovery Laboratory
[36] and AFLOWLIB [37]). The sizes of these databases are in the (NOMAD) [44], Computational Materials Repository (CMR) [45] and
100 GBse1 TB range and growing as they also leverage others, such the Automated Interactive Infrastructure and Database (AiiDA) [46],
as the Inorganic Chemistry Structural Database (ICSD) and the are increasingly providing automated workflows and materials
Cambridge Structural Database (CSD). By contrast, modern day user design infrastructures to facilitate design.
facilities with powerful light and neutron sources, such as the Machine learning tools are being increasingly employed and
Advanced Photon Source (APS) at Argonne National Laboratory, the interfaced into the evolving data pipeline to facilitate property
Spallation Neutron Source (SNS) at ORNL and the free-electron calculations. The associated databases are dynamic, changing day
laser at the Linac Coherent Light Source (LCLS) at SLAC generate by day, and comprise millions of compounds. The ICSD, for
10e100 TBs of data on one sample per beam time. For example, in example, is used extensively to calculate properties of existing
situ high-energy diffraction microscopy experiments at APS, such as structures (and various prototype structures) with different ele-
stretching a sample and studying the evolution of grains, can ments to generate new structures that might be stable. Calculated
involve 54,000 detector patterns and take up to 10e12 h per layer properties in the databases range from magnetic, electronic (e.g.,
using 2048 CPUs on modern architecture with 50 layers needed to band gaps), chemical, structural and thermodynamic, and the range
cover a 3D volume [38,39]. In this case, data collection is typically of materials studied include alloys and metallic glasses, thermo-
slow and very cumbersome, with the biggest challenge being the electrics, magnetic Heusler alloys, strengthening precipitates, pie-
reconstruction of a real-space image from the detector patterns to zoelectrics, battery anodes and cathodes, hydrogen storage
yield materials-specific information. Currently, this reconstruction materials, polymer dielectrics and (more recently) 2D materials.
is performed by solving a forward diffraction model in a brute force These databases increasingly allow users to load their own com-
manner to find the crystal orientations responsible for given pounds for computation and analysis in an ever-expanding
diffraction spots originating from certain grains in the sample [40]. “ecosystem”, clearly changing the manner in which computations
Similar problems emerge in the context of high-resolution electron and materials design will be performed in the future.
microscopy, where the open databases of electronic spectra Several recent studies have claimed to discover new materials
(EELSDB) and atomically-resolved images have recently been using the approaches outlined above. In some instances these
 J.M. Rickman et al. / Acta Materialia 168 (2019) 473e510 479

Fig. 4. The materials data landscape spans a range of data sizes, from those obtained from synthesis and characterization in lab experiments to those assembled via high-
throughput calculations, experiments, and big data accumulated at user facilities. These facilities include the Advanced Photon Source (APS) and the Linac Coherent Light
Source (LCLS) at SLAC. For comparisons, also shown in the log scale plot are the Large Hadron Collider (LHC) and future facility concept, such as Matter Radiation in Extremes
(MaRIE). (P.V. Balachandran and T. Lookman, unpublished, 2017).

discoveries have been validated by experiments, but for many important (as we will see below); however, the community is also
others experimental validation remains an outstanding issue. For beginning to embrace machine learning surrogates as a means to
example, Mounet et al. [47] recently considered the exfoliation of obviate performing exhaustive DFT calculations. A relatively early
108,423 previously synthesized 3D compounds from which they example of this latter procedure is discussed by Saal et al. [36] who
identified 5619 layered compounds. They then used van der Waals employed a decision-tree model to predict formation energies via a
DFT to identify 1036 compounds from among this subset that were training set of 4000 ternary formation energies from OQMD, and
exfoliable and, from this latter group, 50 new ferromagnetic and then used the model to predict the energies of 8700 other com-
antiferromagnetic systems, including half-metals and half- pounds to ensure reasonable agreement with DFT and experi-
semiconductors. Some of the magnetic compounds predicted us- mental results (within 0.11eV/atom). From this study, they then
ing this process include BiI3 and FePS3 and, although they validated compiled a stability map for 1.6 million compounds.
their calculations on experimentally synthesized materials, many of Rupp et al. [52] have built efficient machine learning models to
their predictions on 2D layers await experimental validation. predict properties, such as atomization energies, of molecules with
The search for 2D layered compounds with electronic, me- accuracies approaching those of electronic structure calculation. In
chanical and optical properties, quite distinct from their bulk their approach they employ a ‘‘Coulomb’’ matrix representation for
counterparts, is an active area of research spawned by the Materials the descriptors such that the off-diagonal elements correspond to
Genome Initiative (MGI) with applications in energy storage and the Coulomb repulsion between atoms I and J, while diagonal el-
conversion, such as solar water splitting [48]. Many of the crystal ements encode a polynomial fit of atomic energies to nuclear
structures discovered to date can potentially serve as templates for charge. Using a Kernel Ridge Regression model trained on 1000
structural determination via genetic algorithms [49] and, for new molecules with 3e5 heavy atoms and randomly selected larger
databases such as the Computational 2D Materials Database (C2DB) compounds covering the entire energy range, they applied their
[50], can be used to target applications related to spintronics, (opto- model on a test set of 6000 molecules to show that their repre-
) electronics and plasmonics. sentation yielded errors of roughly 15 kcal/mol, or 1% on the scale of
An example of a high-throughput study that predicted and energies. The same group of researchers has recently also shown
subsequently synthesized new Heusler alloys is that of Sanvito et al. [53] that a machine learning scheme with DFT accuracy can be
[51]. These workers started with a database of Heusler alloys con- constructed for solids to calculate density-functional theory quality
taining over 236,000 compounds, from which they down selected formation energies of all ~2  106 pristine ABC2 elapsolites with an
almost 248 intermetallics that were thermodynamically stable (i.e., accuracy of 0.1 eV/atom for a training set consisting of 1000 elpa-
from an energetic convex-hull analysis, as described below). Of solites comprising main-group elements. These studies emphasize
these alloys, 20 were found to be magnetic, and the authors pro- an important research direction for the development of machine
ceeded to synthesize the ferromagnet Co2MnTi and the antiferro- learning models, based on an appropriate representation, as sur-
magnet Mn2PtPd. An interesting aspect of their work is that they rogates for DFT-based calculations.
also considered a separate experimental database of about 40
Heuslers with known magnetic ordering temperatures, Tc. By using
descriptors such as the volume, magnetic moment and number of 2.3.1. Search for thermoelectric and battery materials
valence electrons, they constructed regression models for Tc to Over the last few years there has been considerable interest in
predict the ordering temperature of ferromagnetic Co2MnTi, which predicting high-efficiency, thermoelectric materials using high-
is relatively high and consistent with experiment. This coupling of throughput screening methods. For example, Carrete [11] used a
high-throughput methods and experimental data to discover new database of almost 80,000 half-Heuslers to identify 75 stable half-
materials is a theme that will no doubt become increasingly Heuslers that had the figure of merit, ZT, value greater than 0.5 at
room temperature, implying that these alloys were competitive
480 J.M. Rickman et al. / Acta Materialia 168 (2019) 473e510

with other thermoelectrics. The predicted alloys included those screening to experiment, and one usually accounts for a degree of
such as BiBaK, SbNaSr, AuAlHf, CoBiZr and RhSnTa. Recently, more metastability when predicting synthesizability. In this regard, using
refined calculations of electronephonon coupling in half-Heuslers the Materials Project database, Sun et al. [59] studied the departure
suggest that the electron relaxation time varies significantly with of system energetics from the convex hull as a function of chem-
chemical composition and carrier concentration, indicating the istry and composition and proposed considering those materials
need to go beyond the usual approximations to screen and optimize both on, and a certain distance above, the hull as potentially useful
materials’ composition and carrier concentration. Yan et al. [54] compounds.
have proposed a semi-empirical descriptor in terms of carrier Several recent studies have revealed interesting comparisons
mobility, effective mass, and lattice thermal conductivity for pre- between predictions made from different high-throughput studies
dicting thermoelectric materials and described its use in high- and between high-throughput predictions and those made by
throughput calculations. They show how known thermoelectrics, learning from experimental data. Such studies also suggest inter-
such as PbTe, Bi2Te3 and Yb14 MnSb11 are predicted using the esting approaches to exploit databases for materials discovery via
descriptor, and also suggest compounds that may be worth pur- machine learning, beyond simply using repositories for materials
suing, including Cu2 CdGeTe4 and KSnSb. Moreover, this group of screening or property prediction. In particular, a comparison of
researchers has developed a Thermoelectric Design Lab, a database high-throughput calculations from three studies for new half-
interface, that calculates thermoelectric properties and perfor- Heusler compounds by Legrain et al. [60] found significant in-
mance rankings based on a metric [55] that offers an assessment of consistencies with respect to whether a given chemical composi-
the material properties influencing ZT. However, the point should tion would be found in a half-Heusler structure or not. The chemical
be made that machine learning methods that rely on statistical compositions they considered were taken from the ICSD and a set
models connecting possible relevant features to experimentally of prototypes generated by a combinatorial decoration of the three
determined ZT are limited by the fact that most materials reported allowed crystal structures. They also found that the high-
have moderate to high ZT; this implies that the training data used to throughput predictions were quite inconsistent with the experi-
develop the predictive models is restricted in range and, therefore, mentally observed structures; however, machine learning models
not well-balanced. trained on the experimental data had cross-validated predictions of
The search for materials for battery applications is also a very half-Heusler alloys with 91% accuracy, and with 99% accuracy alloys
active research area. For example, Kirklin [12] exhaustively that were inconsistent with this structure.
enumerated 515 thermodynamically stable lithiation reactions of Similarly, Balachandran et al. [61], in comparing the predictions
transition metal silicides, stannides and phosphides to compute the of new perovskites using OQMD, found differences with known
cell potential, capacity and volume expansion for each of these data and a similar degree of success with the cross-validated pre-
reactions. The most promising anode compositions they discovered dictions of machine learning models. They emphasized that part of
included CoSi2, TiP and NiSi2, with properties superior to carbon. the reason for these disparities is that the convex hull analysis and
A system holding promise for Li- anode applications is Li-Si. machine learning models are focused on two different quantities,
Tipton et al. [56] used ab initio molecular dynamics simulations the former predicting the zero-temperature mechanical stability of
to model the structure of amorphous Li-Si as a function of the crystal structure and the latter predicting alloy formability
composition, together with a genetic algorithm coupled to density- (given the training data of materials formed and measured). They
functional theory to search the Li-Si binary phase diagram for demonstrated that if a degree of metastability is added to the
metastable crystal structures. They determined the energetics of convex hull analysis, then the OQMD stability predictions show
the stable and metastable Li-Si phases likely to form during the moderate (67%) agreement with compounds known to have been
charging and discharging of a battery, and identified Li5Si2 via a formed. Nevertheless, such an analysis is of value to guide experi-
convex hull analysis as a previously unreported member of the Li-Si mentalists because, if machine learning predicts that a compound
binary phase diagram that is stable at 0 K with respect to previously is formable and OQMD predicts that it is stable, this compound is
known phases. In the search for novel electrolytes for electrical more likely to be formable experimentally as a perovskite relative
energy storage systems, Cheng et al. [57] discussed a hierarchical to other machine learning predicted prototypes. In this case, a total
method to screen multiple properties using quantum calculations of 87 new perovskites, including 6 in the cubic phase, were pro-
(Fig. 5). This approach down-selects a pool of candidates based on posed for synthesis. These studies emphasize the need to address
properties such as redox potentials, solvation energies and struc- and control issues associated with the physical and computational
tural changes of over a thousand organic molecules for non- origins of information intrinsic to high-throughput studies, such as
aqueous redox flow batteries. The hope is that this paradigm, thermodynamic stability, entropic contributions and the role of
typical of high-throughput calculations, will have significant different functionals and pseudopotentials in calculations.
impact in the search for new materials with targeted properties.
2.3.3. Search for perovskites and glasses
2.3.2. Stability in high-throughput computations The search for perovskites with tailored properties and appli-
In high-throughput screening studies, stability is based on the cations via high-throughput screening, especially for solar cell
convex hull analysis [58] in which the total energy of a compound is needs, is an active area of research with a large compositional space
calculated in a fixed stoichiometry (say ABO3), and the structural (>106) having different structural motifs. In this regard, the search
type yielding the lowest energy based on alternative crystal for lead-free inorganic/organic hybrids comprising organic and
structures of the target compound ABO3 and its decomposition inorganic constituents, such as CH3 NH3, HC(NH2)2, Cs, Rb, Sn, I and
products, such as A þ Bþ3/2O2, or AO þ BO2, is selected. Thus, the Br, in layered, double perovskites and metal-deficient forms have
stability of a chemical composition for a particular structure is recently been reviewed by Chakraborty et al. [13]. These searches
measured by the energy difference between it and the convex hull, are motivated by the observed shortcoming that lead-free perov-
and so a stable compound is located on the hull whereas a meta- skites exhibit poor power conversion, with efficiencies in the low
stable compound will sit above the hull. It should be noted that single digits. However, a recent study of single crystal CsSnI3 [62]
some experimentally known compounds are, in fact, metastable, as has reported smaller defect densities, improved diffusion lengths
can be seen from kinetic considerations. Thus, care should be and carrier life times, and a predicted efficiency of ~ 23%. To
exercised when comparing the results of high-throughput expedite the search for viable perovskites, machine learning is also
 J.M. Rickman et al. / Acta Materialia 168 (2019) 473e510 481

Fig. 5. A schematic of the hierarchical approach to high-throughput quantum mechanical computations to down select promising electrolyte candidates for electrical energy
storage applications based on property evaluations [reproduced from Ref. [57]].

being increasingly combined with DFT calculations and databases metastability (100 meV/atom). Their analysis identified CsNbO3,
within a target-driven approach. For example, Lu et al. [63] trained RbNbO3 and CsTaO3 as candidates warranting further study. It is
a machine learning model for band gaps using 212 reported hybrid worth noting that, by contrast with these somewhat idealized ef-
perovskites and features such as tolerance factor, octahedral factor, forts, the real materials problem is quite complicated. To begin
metal electronegativity, and polarizability to predict the bandgaps with, one is dealing with solid solutions for which robust and
of 5158 unexplored possible hybrids. They then down select six reliable DFT studies are not easily performed; moreover, the high-
stable Pb-free compounds, including NH4InBr3, as promising solar throughput studies cited above are all for single perovskites. Sec-
cells materials with appropriate bandgaps. These authors make the ond, the MPBs are very sensitive to the choice of the two end
point that, by contrast with much of the high-throughput work compounds used to form the solid solution; a change of 0.01 in the
underway in which the whole chemical space is searched at the DFT compositions m and n in Ba(Ti1-mXm)O3-x(Ba1-nYn)TiO3 changes the
level, combining high-throughput and DFT calculations reduces curvature of the MPB and hence affects the piezoelectric response.
computational resources as only the most promising compounds Finally, for the BaTiO3-based system, only 20e30 well character-
must be searched at the DFT level. ized and synthesized compounds are available out of a possible
An example that contrasts a high-throughput DFT calculation 1000 odd solid-solutions or phase diagrams [67]. Hence, this en-
with learning from experimental data is the search for lead-free terprise requires methods that allow one to learn from relatively
piezoelectrics, which we will discuss in greater detail in the next small amounts of experimental data to guide the discovery process,
section. More specifically, Armiento et al. [64,65] and, very recently, which we consider below.
Barr et al. [66] used DFT to enumerate candidate materials (~3500 With regard to glasses, an interesting recent study, emphasizing
in Ref. [64]) with the perovskite ABO3 structure, substituting one of the importance of good descriptors, combines thermodynamically
63 odd elements for the cations A and B in the periodic table. This competing ordered structures with calculated data available in
list was further culled by down selecting those candidates likely to AFLOWLIB to predict the glass-forming ability (GFA) of metal-alloy
be insulators via the addition of a threshold for the Kohn-Sham systems. Perim et al. [68] used statistical approaches to propose
band gap of 0.25eV, thereby reducing the total to a manageable that ordered phases, which have similar energies, mutually
number (~100). By then demanding that the energy difference compete during solidification, thereby frustrating crystal nucle-
between the tetragonal and rhombohedral structures be suffi- ation to promote glass formation (Fig. 6). They used the dispersion
ciently small (<0.5eV) to allow ease of distortions across the mor- of formation energies of the ordered structures as a descriptor of
photropic phase boundaries (MPB) separating these two phases, a GFA to predict 73% of the glass-forming compositions for a set of 16
final list of about 50 possible single-phase perovskites was pro- experimentally characterized alloy systems. Their findings indicate
posed. Barr et al. [66] also sought to identify Pb-free piezoelectrics, that the formation of metallic glass phases could be much more
but with additional screening and constraints to preclude A-B common than currently thought, with more than 17% of binary
combinations that could form ternary compounds with Pt (but that alloy systems as potential glass formers. The issues related to phase
were also stable in a Pt environment). The motivation for these stability also arise in Section II.3.2 and have also been discussed
constraints was to avoid the possibility of forming secondary there.
phases with Pt electrodes while also permitting a degree of
482 J.M. Rickman et al. / Acta Materialia 168 (2019) 473e510

Fig. 6. Prediction of glass-forming ability (GFA) of metal-alloy systems from the spread of formation energies of relevant ordered structures. Different structural phases with similar
energies compete during solidification, frustrating crystallization and promoting glass formation. A broad (narrow) distribution of energies implies a low (high) GFA (a) Energy-
composition plot of CuZr showing the minimum energy surface (convex hull). (b) Structural phases (with composition Cu 0.5Zr0.5) close in energy causing frustration during so-
lidification. (c) Illustrations of two examples of energy distributions characterizing low and high GFA, respectively (reproduced from Ref. [69]).

2.4. Inverse design framework) is simply a necessary step in carrying out the targeted
exploration of a materials design space (inverse modeling). A
The possibility of inverse design for targeted materials proper- schematic illustrating Zunger's perspective is shown in Fig. 7.
ties was discussed by Jain et al. [70], who suggest that at the coarse Finally, we mention that genetic algorithms have been exten-
level inverse approaches target a class of morphologies that would sively used to optimize specific target properties. These algorithms
be optimal (within known material property constraints) for real- can be efficient in searching for materials with desired properties
izing specified macroscopic properties. Once these morphologies when compared to random searches or those based on chemical
are known, the aim then is to design fabrication or assembly routes rules [72]. For example, Mannodi-Kanakkithodi et al. [73] obtain
with appropriate material precursors for realizing the required specific n-block polymers that simultaneously have a desirable
structures. They therefore propose a hierarchical framework, as dielectric constant and bandgap, without having to consider all
exemplified by topological optimization, to discover new micro- possible dielectric polymers. The idea here is to start with a random
structures consistent with acoustic, nanophotonic and photovoltaic initial population of n-block polymers (for any given n) and let
behavior. There have been notable successes with this approach, them evolve (in terms of constituent blocks and their neighbors)
namely nonlinear constrained optimization to design surface tex- based on a genetic algorithm to yield a set of polymers with
tures of thin-film solar cells for increased photon absorption [70]. properties closest to the stipulated targets.
Recently, Zunger [71] provided his perspective on the relative
merits of direct versus inverse design for functional materials in
which, by contrast with the direct (forward-modeling) approach, 2.5. Adaptive learning for materials discovery
he suggests that inverse modeling can occur via three routes: 1.)
searching for structures, such as superstructures with target func- Given that in many materials science problems the amount of
tionalities (e.g. a direct-gap semiconductor composed of a specific the quality data is limited, we consider below approaches to
sequence of indirect-gap Si and Ge building blocks), 2.) searching learning from relatively small amounts of data to guide future ex-
for chemistries with given structures (e.g. a family of AMX3 halide periments. In other words, we review methods and applications
perovskites), and 3.) exploring missing compounds (e.g. missing that guide the optimal experiments to be performed sequentially to
half Heusler alloys). The point should be made that simulations find materials with a targeted response [74e77]. This effort thus
connecting structure to properties (forward modeling in Zunger's makes contact with aspects of experimental design, a well-
developed area within statistics [77], and surrogate-based
 J.M. Rickman et al. / Acta Materialia 168 (2019) 473e510 483

Fig. 7. A perspective on direct and inverse approaches to materials design, as suggested by Zunger [71]. The bottom graph shows the possible compounds that can be formed from
components with atomic numbers ZA and ZB. The blue squares represent existing compounds of known composition and structure. The question-mark area represents chemically
plausible, but as yet unreported compounds. In the direct approach to materials design, all compounds depicted in the bottom graph are investigated and, for each compound, the
presence and extent of a certain property and/or functionality are calculated. In functionality-directed material discovery or inverse design, only those compounds with the target
(or maximal) functionality are investigated (reproduced from Ref. [71]). (For interpretation of the references to colour in this figure legend, the reader is referred to the Web version
of this article.)

modeling [78], the latter using ideas based on the value of infor- or calculation based on certain criteria. The new results augment
mation and selection strategies for policies [79]. The aim here is not the training dataset, and the loop repeats itself until the desired
to discover immediately a material having optimal properties, targets are met. The problem is to optimize (say maximize for our
which is unrealistic. Rather, one searches iteratively to find that purposes) f ðxÞ; that is, to find the design or material represented by
part of the parameter space where a material with the desired the features, x, that maximizes f . The idea is to use some surrogate
property may be found, thereby reducing the requisite number of or inference model (off-the machine learning tools such as support
experiments. This approach is therefore a departure from high- vector or kernel ridge regression, decision trees or gradient tree
throughput screening motivated by electronic structure calcula- boosting) to make the ‘best’ prediction.
tions in which the emphasis is on down selecting the best candi- There are, however, very few examples that one can cite of new
dates for further studies. materials synthesized, characterized or predicted using only this
Before assessing the implications of limited data in materials approach. There are at least two reasons for this situation. First, the
applications, it is useful to examine first the field of bioinformatics output from the surrogate prediction is not necessarily optimal as it
where the problem of small sample sizes and a large descriptor or ‘exploits’ the model's prediction. Second, one can often be trapped
feature space (in the hundreds and thousands) is well known [80]. in a local minimum of a high-dimensional feature space if the cost
For example, classifiers must be trained to predict different types of function is not convex, and so this ‘best value’ prediction is often
breast cancers using gene signatures as the features, and there is a not what is desired. Hence, it is essential that there is a means to
tradition in that field of using probabilistic Bayesian methods that ‘explore’ the feature space adequately by choosing a better next
incorporate prior knowledge. It has been realized that such an experiment rather than simply using the best output from the
approach has predictive merit for developing classifiers and re- surrogate model, which just interpolates between data points. At
gressors and their associated errors, as opposed to using standard this juncture experimental design enters the picture and, in its
cross validation methods that are often not reliable for small data classical formulation, one studies the behavior of a system in which
sizes. By contrast, in materials informatics, only recently have there are variations in data controlling inputs and outputs [77].
workers undertaken in-depth analyses of regressor and classifier From this perspective, factorial and composite designs are then
performance on materials data sets, and the importance of un- used as heuristics for exploring the relationship between input
certainties in predictions is also only beginning to be appreciated factors and response variables. The idea is to obtain the in-
[81e83]. terrelationships among the various factors and their impact on the
We review below aspects of materials discovery that are seldom response to prioritize the next set of experiments. One can be
addressed, namely guiding the next experiments (or calculations), guided here by the notion of the value of the information (or cost of
especially if they are expensive in terms of costs and/or time. This uncertainty) that is gained (or reduced) by observing a particular
analysis goes beyond mere regression or classification studies, experiment or data point. The possibilities of choosing certain ex-
which are important in training inference models from data, to periments can then be ranked by the expected value of the infor-
address the problem of designing future experiments that identify a mation they provide, thereby facilitating experimental
material with a given label or property. Fig. 8 shows the adaptive prioritization [74].
design or active learning approach that contains aspects of optimal
experimental design beyond merely training a “surrogate” or 2.5.1. Bayesian Global Optimization
inference model. A dataset of features and properties serves as Consider again Fig. 8 which illustrates the adaptive design or
input to a surrogate model, which can be based on machine active learning approach to materials discovery. The two-step
learning related methods and/or a reduced order or physics-based process comprising the active learning loop in the figure involves
model. The predictions and their uncertainties then flow to an 1.) the construction of the surrogate model, and 2.) the concept of a
experimental design component that selects the next experiment utility function, previously introduced in decision theory and
484 J.M. Rickman et al. / Acta Materialia 168 (2019) 473e510

Fig. 8. The adaptive design loop that is the basis of the surrogate-based modeling approach to optimization. A dataset of features (or descriptors) established using materials
knowledge and properties is typically the input to a surrogate model, which can be constructed using machine learning methods (e.g. a Gaussian process) and/or based on a reduced
order or physics model. The predictions with uncertainties from the surrogate model serve as input to an experimental design component that selects the optimal candidate for
subsequent experiments or calculations by maximizing the expected utility function. This function encodes the basis for the decision-making process (from Ref. [84]) in the Bayesian
Global Optimization. The new results augment the dataset, and the loop repeats itself until the desired targets are met.

related to the value of information (and hence uncertainties) used have they been applied to problems related to materials discovery.
to make optimal sampling decisions [85,86]. We briefly describe The utility function, uðxjDÞ, is defined so that its maximum over all
several of these functions, such as the widely-studied expected the data in the sampling domain, that is x* ¼ argmaxx uðxjDÞ ;
improvement function, that provide selection criteria. We note that produces a (potentially) high value of the objective at which to
this two-stage approach is also known as Bayesian Global Optimi- evaluate the function and thereby recommend the next experiment
zation (BGO) (Fig. 8) if the surrogate model is replaced by a or calculation. This recommendation is the decision-making step
Gaussian process. that directs future sampling. Upon augmenting the data set with
A Bayesian approach for the surrogate model, where f ðxÞ is new information obtained from experiment or calculation, the
sampled from a prior distribution that assumes smoothness and surrogate model, or GP, is retrained and forms part of an active
locality, has been found to be quite powerful in locating the learning loop that is the basis of BGO (Fig. 8).
extrema of unknown functions, especially when they are costly to The utility function may be selected from a range of options,
evaluate and if information on their derivatives and convexity including those based on minimal or maximal variances associated
properties is not available. Moreover, the assumption of a Gaussian with the predictions for all allowed possibilities that constitute the
prior has been demonstrated to be quite efficient in guiding search space. Other options include: lower or upper confidence
successive sampling. In brief, the Bayesian approach can be de- bounds, criteria based on the probability of improvement (P[I]) and
scried as follows. One begins with the assumption that D ¼ its expectation known as expected improvement E[I], expected
fðxi ; yi Þ; i ¼ 1; N g, are a set of observations from some probability quantile improvement and knowledge gradient (KG), sequential
distribution f ðxÞ with a Gaussian noise term,N ð0; sÞ, having zero kriging optimization and the modified objective cost of uncertainty
mean and standard deviation, s, that is associated with the pre- (MOCU) [89]. A Gaussian process for the surrogate, combined with
dictions of yi so that yi ¼ f ðxi Þ þ εi : Then, using Bayes’ rule, the E[I] for the utility function, is an efficient method for global opti-
posterior distribution is given by Pðf jDÞfPðDjf ÞPðf Þ, where the mization (EGO) and has been applied extensively [92]. For the
posterior gives the updated distribution or belief for the unknown purposes of illustration, we briefly consider P[I] and E[I], as they
f ðxÞ in terms of the probability Pðf Þ (i.e., prior information as- have been used in several materials-related examples, and because
sumptions on f ðxÞ) combined with the conditional probability other utility functions can be related to them.
PðDjf Þ (i.e., the data likelihood). For a Gaussian process, which is a The improvement-based function, I, was initially suggested by
distribution over functions specified by a mean function and a Kushner [90e92] to maximize the probability of improvement over
covariance, it can be shown using standard formulae that Pðy0 jD; x0 the best outcome observed in the training data, f ðxþ Þ. That is, if I ¼
P
Þ ¼ N ðm; SÞ, where m; are the mean and covariance matrix, max ðy  f ðxþ Þ; 0Þ is the improvement for a proposed feature, x,
0 0
respectively, of the Gaussian distribution and y ; x are the estimates where it is assumed that the possible values of y ¼ f ðxÞ are nor-
0
of the mean of y at a given x : The predicted mean and variance of mally distributed according to probability PðyjxÞ; then the Ex-
the Gaussian process is then used as the input to the utility func- pected Improvement E½I possible is given by the expectation
tion. If the Gaussian process is not used as a surrogate, the mean
and variance can be estimated from the data by the method of 2 2 3
f ðxþ Þ
    ð   
bootstrap sampling with replacement to mimic the properties of 6 6 7
the population drawn from the original data sample [87]. E½I ¼ E6
4 max6 y  f x ;0 7 ¼
4
þ
5 y  f xþ :PðyjxÞ dy:
∞
2.5.2. Utility functions
(1)
The second step of the active learning loop is the choice of the
utility function and, while these ideas date back several decades to It can be shown that E½I can be expressed as E½I ¼ s½fðzÞ þ
þ
aspects of information and decision theory [79,88], only recently zFðzÞ, with z ¼ msm and where m and s are the mean and variance,
 J.M. Rickman et al. / Acta Materialia 168 (2019) 473e510 485

respectively, for y, and mþ is the mean for f ðxþ Þ. The functions fðzÞ output current in an optoelectronic device [4]. In addition, Bala-
and FðzÞ are the standard normal density and the associated cu- chandran et al. [97] compared the performance of the loop on a
mulative distribution functions, respectively. If we consider the computational data set of elastic moduli of 240 hexagonally layered
limiting cases of model uncertainty as s/0, then E½I/m  mþ . M2AX compounds assembled by Cover et al. [98], where X refers to
Therefore, the candidate material, x* , is chosen to be the one with atoms located in the edge-connected M octahedral cages and A
the largest m or the “best” prediction; that is, we exploit this pre- denotes atoms located in slightly larger right prisms. They
diction in local regions of the search space where the model is likely compared multiple selection strategies and demonstrated that
to yield good predictions near the optimal solution. For s/∞ (or, in relatively few new measurements (~35) are needed on average to
practice, large), E½I/s, so that x* is the candidate with maximum find a target compound in the data set for a relatively small initial
variance. If s and m are both large, then E½I is also large, implying (training) dataset (Fig. 9). The authors found that the performance
that the favored candidate x* is the one with the largest predicted m of random selection and the exploitation of model predictions are
but with the greatest uncertainty. This interplay between s and m is inferior to those associated with the EGO and KG on this data set.
characteristic of the exploitation e exploration trade-off, which is a Recently, Ling et al. [83] examined four different data sets, including
feature of algorithms based on so-called selection and ranking computational data for magnetocalorics, superconductors, ther-
problems, such as the multi-armed bandit problem in which moelectrics and steel fatigue strength, with the aim of property
limited resources must be allocated among alternative choices to optimization. In particular, they evaluated the performance of
maximize the expected gain [74]. There are several ways to solve several strategies in terms of number of evaluations required to find
the multi-armed bandit problem, including the no-exploration and the best property in the data sets. Their comparisons involved
exploration-at-random algorithms. For example, the well-known calculating uncertainties based on the addition of a bias term (or an
ε-greedy algorithm [93] takes the best action (“greedy”) most of ensemble uncertainty for random forests), drawing from a
the time, but picks a random option with a small probability ε on Gaussian process model (using the code COMBO [99]) and random
some occasions. selection. It was found that random selection was the inferior
Several other utility functions have been employed in materials strategy, while the performance of the other strategies was dataset
prediction. For example, the knowledge gradient, KGðxÞ [89], is dependent.
based on Eq. (1) but incorporates noise and involves a wider class of As noted above, materials informatics has been employed to
materials. This is because the definition of mþ involves all possible accelerate materials discovery. For example, Xue et al. [84]
materials, not just those forming the training data, including those demonstrated how one can accelerate the discovery of new,
predicted in the search space. The material then chosen corre- multicomponent NiTi-based alloys having the minimum thermal
sponds to the candidate x* ¼ argmaxx KGðxÞ, and KGðxÞ reduces to hysteresis. They focused on the B2 to R transformation in these
E½I in the absence of noise. Thus, the knowledge gradient method shape memory alloys (SMAs), which are known to have smaller
aims to find the sample that most improves the model. Other utility transformation strains and hystereses as compared to the B2 to
functions related to KG and EGO include the Sequential Kriging orthorhombic or the B2 to monoclinic transformations in the family
Optimization (SKO) and the upper confidence bound, the latter of compounds given by Ni50-x-y-zTi50Cux Fey Pdz (where 50 - x - y - z
taking a linear combination of exploitation (m) and exploration (s)  30, x  20, y  5 and z  20). For training purposes, the authors
such that the admixture varies with the number measurements assembled a database of their own measurements for compounds
already made. A comparison of the relative performance of these synthesized in an arc-melter under the same processing conditions
functions on Gaussian data [94] shows that the SKO works and protocols. Each loop comprised four independent composition
extremely well in practice. Finally, a promising, recently-introduced selections for synthesis and characterization by differential scan-
function is the Mean Objective Cost of Uncertainty (MOCU), ning calorimetry and x-ray diffraction. The models consisted of
formulated for complex dynamical systems and developed in the support vector regression and Gaussian processes, together with
field of bioinformatics in connection with the design of gene reg- EGO that maximizes the E[I]. They initiated the loop with 22 well-
ulatory networks [95]. Unlike traditional measures, such as Shan- characterized training datasets from a palette of 800,000 allowed
non or relative entropy (KL or Kullback-Leibler divergence) or compositions with a composition control of 0.1%. The compound
quadratic loss to reduce the overall variance in a system to monitor with the best performance was found to have a thermal hysteresis
information gain, MOCU uses an objective-based uncertainty of 1.84K, as measured by the peak-to-peak interval in the heat flow
scheme in which new measurements are selected to reduce un- as a function of temperature on cooling and heating. By succes-
certainty and identify materials with desired properties [82]. More sively projecting the objective to a lower-dimensional feature space
specifically, MOCU compares performance degradation between an and carrying out DFT calculations, one of the conclusions of this
optimal design (based on incomplete prior knowledge and data) work was that the composition of Fe in the new alloy was a key
and one made with full knowledge of the system. The next exper- controlling factor leading to low hysteresis values.
iment that reduces the uncertainty the most (i.e. the one expected Materials informatics is also a useful tool for guiding future
to minimize the variance in the posterior distribution the most) is experiments. Consider the prediction of transformation tempera-
then selected. tures that often determine the working window for device appli-
cations. Xue et al. [100] employed a polynomial model with
2.5.3. Applications to materials problems features that capture chemical bonding and atomic size and per-
There are several examples in which the methodologies out- formed regression and design using EGO, KG and maximum vari-
lined above have been used to discover new materials and accel- ance to predict a transformation temperature in an alloy. Their
erate computational codes. We begin with applications of active model for the transformation temperature depends linearly on the
learning strategies. As a first example, Chen et al. (2015) [96] weighted average of the valence electron number and Pauling
employed a kinetic model and a Bayesian inference based KG model electronegativity, and quadratically on Waber-Cromer's pseudo-
to examine the stability of an emulsion, whose release is triggered potential radius of the constituents of the alloys. Using their rela-
by the excitation of gold nanoparticles functionalized on the sur- tionship, they formulated a simplified Landau-type model to
face of oil droplets. Their goal was to obtain a system that had the understand the controlling factors that dictate transformation
targeted controlled release properties via a set of tunable model temperatures in NiTi-based alloys. They predicted that Ti50Ni25Pd25
control parameters. KG was similarly used in maximizing the would have the highest transition temperature of 189.56  C, which
486 J.M. Rickman et al. / Acta Materialia 168 (2019) 473e510

Fig. 9. Test case showing surrogate model and utility functions for a computed data set of 223 M2AX phase compounds [96] with elastic moduli. (a) The crystal structure and
chemical search space of M2AX phases. (b) The relative performance of three regressors as a function of the number of measurements or size of the training data set, showing that
the regressor SVR with a radial basis function performs much better than SVRlin and better than GPM if the size of the training set exceeds 100. (c) For SVRrbf the number of new
measurements (or iterations of the loop of Fig. 8) is plotted as a function of the number of initial measurements in the training set using different selection strategies. With fewer
initial measurements, EGO, KG and Max-P (maximum in P[I]) discover the optimal composition with largest modulus with fewer new measurements than Max (largest mean) and
Max-A (alternately select largest value and most uncertain). Choosing measurements at random gives the worst performance. (reproduced from Ref. [97]).

compares well with the measured value of 182.89  C. experiments involving hundreds of data points based on parallel
It has also been possible to employ informatics to identify ma- synthesis using magnetron co-sputtering.
terials having desired mechanical properties or structures. For Adaptive design can be used to identify models that best
example, Dehghannasiri et al. [82] used the reduction in MOCU, the describe a dataset. To infer the subsurface properties from the
mean objective cost of uncertainty, to identify subsequent candi- surface of a material in a thin film deposited on a substrate, for
dates within a phase-field (or Landau) model of FePd, an SMA, to example, Aggarwal et al. [102] used a Cahn-Hilliard model, together
minimize mechanical dissipation by selecting dopants and their with Bayesian Global Optimization, to choose experiments that
associated potencies. Although little studied in the materials provide maximal information about parameters in the model for a
domain, this method has potential advantages in providing a robust given experimental budget. They maximize the expected utility of
means to incorporate overall uncertainties in an experimental the KL divergence from the posterior to the prior distribution for
design. Recently, Ward et al. [101] considered a slightly different the choice of utility function. They show that the information
strategy in a search for ternary bulk metallic glasses by starting gained in the optimal experiment, with two values of the param-
with data from the literature on glass forming ability related to eters, is comparable to the information gained from the experiment
amorphous ternary systems of the type Co-V-Zr. For this purpose, with eight randomly selected values of the same parameters. They
they used several models, including classification, whose pre- employ a similar approach to study the connection between metal-
dictions they essentially exploit in a staged approach to account for metal interfaces and the trapping of helium (He) impurities, given
synthesis dependence. They conducted high-throughput the impact of He on the performance of nuclear energy materials.
 J.M. Rickman et al. / Acta Materialia 168 (2019) 473e510 487

However, in this case they employed the expected KL divergence to temperatures (898K for {FeCo}).
discriminate among models, given parameters related to experi- Several recent studies have focused on learning from data
ments, observed data and competing models. These examples generated using phenomenological codes. For example, Wang et al.
illustrate the use of adaptive design to find the best pair of exper- [106] considered optimizing interphase properties in polymer
iments for the parameters of a given model, and to search for an nanocomposites by minimizing the difference between the pre-
experiment that can optimally select among competing models, dicted bulk property of a nanocomposite and that from the
where each model has a distinct form and a distinct set of uncer- experimental data. For this purpose, they used a Gaussian Process
tainty parameters. (GP) model as the surrogate in conjunction with EGO. Their
One of the first applications of design to ferroelectrics was the approach was tested on simulations of dielectric and viscoelastic
use of Bayesian inference in conjunction with a Landau-Devonshire properties in nanocomposites. In another example, namely the
model for finding Pb-free BaTiO3-based piezoelectric solid solu- design of light emitting diodes (LEDs), it is desirable to maximize
tions with MPBs that showed little temperature sensitivity [67]. electro-luminescence efficiency at high current densities. Leduc
The family of compounds considered belong to the class x(Ba1m- et al. [107] considered an LED with five quantum wells where the
Cam)TiO3Ba(ZrnTi1n)O3 (xBCTeBZT) and x(Ba1m Cam) indium compositions are the inputs (or features) and they opti-
TiO3 Ba(Snn Ti1n)O3 (xBCTeBTS), and the aim of the study was to mized the quantum efficiency of the wells. They used a Gaussian
find the optimal m and n such that the curvature of the MPB for the process and EGO to show how rapidly the GaN-based LED struc-
ceramic was minimized. The equations for the phase boundaries tures can be constructed to delay the onset of “droop”, the drop in
served to constrain the search space, which consisted of 1200 efficiency across the wells, by spreading the carrier concentrations
possible phase diagrams with different endmembers (or a evenly in the active regions of quantum wells. Their design effec-
maximum of 18000 possible compositions), and the experimental tively guides the selection of the next LED structure to be examined
data consisted of 19 well-characterized phase diagrams of syn- based upon its expected efficiency as well as its model uncertainty.
thesized solid solutions. Using features that coupled the atomic, In particular, the active learning strategy produces a model that
crystal chemistry and electronic structure properties of the two predicts the results of Poisson-Schro €dinger simulations of devices,
endmembers of the solid solutions, the authors used Bayesian and that simultaneously yields structures with higher simulated
regression, as well as a data-driven approach with EGO, to syn- efficiencies. The method rapidly converges to find a nearly optimal
thesize and characterize the solid solution (Ba0.5 Ca0.5)TiO3- simulated LED efficiency in about 75 iterations, and subsequent
Ba(Ti0.7Zr0.3)O3. By studying correlations in the experimental data, iterations result in little improvement with attendant decreases in
Xue et al. [103] also show that the ratio of unit cell volumes and the model uncertainty.
ratio of ionic displacements of the tetragonal and rhombohedral As noted above, most studies have so far focused on optimizing
ends influence the verticality of the MPB in both BaTiO-based and single objectives whereas materials problems often have
Pb-based systems, and therefore can serve as excellent descriptors. competing objectives. Thus, the ideas outlined here have also
As these quantities are not readily accessible for multi-dopant recently been applied to multi-objective optimization so that the
systems, the ratios of ionic radii and effective nuclear charges be- next material identified in an optimization is chosen to improve on
tween the two ends serve as surrogates for developing Pb-free pi- an existing Pareto front. In this regard, the generalization to multi-
ezoelectrics with high-temperature insensitivity. objectives of the surrogate-based optimization approach for single
Recently, Yuan et al. [104] showed how an optimal experimental objectives was considered by Keane [108] and Svenson and Santner
design approach leads to the piezoelectric (Ba0.84Ca0.16)(Ti [109]. Gopakumar et al. [110] applied these methods to several data
0.90Zr0.07Sn0.03)O3 having the largest bias-induced strain of 0.23% sets already discussed, including SMAs, M2AX phases and piezo-
(unipolar strain 0.19%) under a field of 20 kV cm1 in the barium electrics, to assess the behavior of different selection strategies in
titanate family. More specifically, they compared the results of the two-dimensional space of objectives. Recently, a variation on
employing EGO, pure exploitation, maximum variance and random this approach in terms of the hypervolume indicator was intro-
selection over five design loops using their own training data set of duced for multiobjective optimization [111e113]. The hypervolume
61 synthesized and well-characterized compounds from a potential is a measure of the size of the space enclosed by all solutions on the
search space of 605,000 compounds. Using support vector regres- Pareto front and a reference point given by the user. The expected
sion, in conjunction with bootstrap sampling, they found that EGO improvement in hypervolume is then the gain and is a measure of
was a superior strategy as compared to the others. Moreover, it was the closeness of a candidate point to the Pareto front. Although not
found that 3 at% Sn composition is optimal in this context as the as well-studied, the approach is quite competitive as compared to
strain decreases appreciably for deviations from this composition. the usual expected improvement criterion on the objectives stud-
By parametrizing a Landau-Devonshire free energy and performing ied recently in materials applications [114,115]. In terms of
phase-field simulations, they also show that the 3 at% Sn compo- computational costs, typical implementations are considerably
sition is associated with ease of switching of polarization domains more resource intensive; however, recent studies have attempted
in the tetragonal phase at room temperature relative to the com- to significantly accelerate the computations of the multidimen-
pound without Sn. Thus, this work demonstrates that chemical sional integrals that arise in computing the expected improvement
substitutions in piezoelectrics can serve as an effective route to in hypervolume [116]. Related ideas are relevant for the problem of
controlling the electrostrain properties of ferroelectric oxides at the multi-fidelity optimization, in which the aim is to predict the
domain level. A two-step strategy involving both classification and behavior of the surrogate model from a large amount of inexpen-
regression was also recently studied by Balachandran et al. [105] in sive, but low-fidelity data combined with a small amount of
the search for high Curie temperatures in Bi-based solutions that expensive, but high-fidelity data. The idea here is to make pre-
also include PbTiO3 (xBi[M1yM2(1-y) ]O3-(1-x)PbTiO3). The classi- dictions that approach the accuracy of the those associated with the
fication first separates one-phase stable compounds from other high-fidelity data. Pilania et al. [117] have recently shown how to
perovskites that are either unstable or contain multiple phases, and apply this method to the case of band gaps evaluated using the
the regression and design guide experiments towards solutions much cheaper LDA method of electronic structure calculation and
with high Curie temperatures. With this approach the authors were the more expensive, but accurate HSC hybrid functional method.
able to find Bi-based perovskites containing novel dopant pairs, They demonstrate that the accuracy of the overall predictions de-
such as {Fe,Co}, {NiSn} and {CoAl} at the B site, with high Curie pends on the relative fraction of the inexpensive versus expensive
488 J.M. Rickman et al. / Acta Materialia 168 (2019) 473e510

data. Although the approaches used so far in materials problems data set. Similarly, a 4D STEM dataset representing a local diffrac-
are largely variants of co-Kriging, recently more sophisticated in- tion pattern at each sample region acquired at (1024  1024)x
formation fusion frameworks involving more than a pair of high- (1024  1024) pixels resolution corresponds to ~4.4 TB. However,
fidelity and low-fidelity models have been used to design optimal advances in high-performance computing and data analysis
microstructures. workflows developed in fields such as astronomy, neutron and x-
Finally, we note that a key result in the field of surrogate models ray scattering and high-energy physics suggest that these data size
and optimization, the so-called no-free-lunch theorem [118], pro- problems, though daunting, are tractable, provided that there is a
vides a sobering constraint on the materials design enterprise. This dedicated data infrastructure capable of storage, metadata searches
theorem states that there is no universal optimizer; that is, there is and discovery. Parenthetically, we note that it is the lack of such an
no guarantee that a model trained on a given data set will also infrastructure that currently limits the development of high-
apply to another data set. Thus, in the case of materials problems information content electron microscopy (e.g., focal series in 4D
where data sizes can be limited, a purely data-driven approach has STEM) and scanning-probe microscopy methods.
its limitations and will not, in general, be robust with well-defined The second challenge that emerges in the context of (hyper-
uncertainties. In many of the studies we have reviewed, including spectral) imaging is the analysis of the data, including both
those leading to new materials, the search spaces are generally exploratory data analysis and data reduction to materials-specific
relatively large and there are no clear stopping criteria. The design information. Indeed, whereas a single image or spectrum can be
loop is terminated once a satisfactory outcome is achieved. Thus, explored and qualitatively analyzed by a human operator, this is not
moving forward, physics-based models will be essential in con- the case for a quantitative analysis of large volumes of data. The
straining the search space and providing adequate prior informa- typical issues that emerge in this context are related to the appli-
tion for robust predictions as new data become available from cability of information-theoretic methods to reduce the data vol-
experiments or calculations. ume to a small number of representations that can further guide
human-based analytics. Moreover, the quantitative analysis of im-
3. Data analytics and image processing ages or hyperspectral data sets based on the known (or postulated)
physics of a problem is often problematic, especially the determi-
The advent of (scanning) transmission-electron microscopy ((S) nation of atomic coordinates in STEM and STM data and peak fitting
TEM), [119] scanning probe microscopies, including both force- and in hyperspectral data sets. In short, it is difficult to convert the
current based techniques, [120] and chemical imaging methods, output of the imaging system to materials-specific information.
such as micro-Raman and time-of-flight secondary-ion mass Finally, the third set of challenges emerges in the context of
spectrometry (ToF-SIMS), has opened the pathway for high- physics extraction. For example, high-resolution STEM data sets
resolution imaging of materials structure and functionality, contain information on atomic positions with ~pm level precision,
generating large volumes of 2D structural, functional and hyper- which in principle is related to the strength of atomic interactions
spectral data. In the field of (S)TEM, for example, the broad intro- and the nature of structural instabilities in the system. Similarly, the
duction of aberration correction in the early 2000s has enabled experimentally observed dopant atom positions and defect con-
numerous advances, including high-veracity imaging of materials figurations are directly related to the thermodynamics of the solid
structure with atomic resolution of imaging-beam sensitive mate- solutions in the material. Thus, the extraction of the underlying
rials and light elements. The proliferation of commercial platforms atomic interactions and macroscopic constitutive relations from
has brought this high-resolution STEM capability to multiple aca- imaging data represents an interesting, albeit very lightly explored,
demic, commercial, and government labs. Perhaps more impor- area of research. It is interesting to note that various approaches to
tantly, this enhancement in resolution has enabled the application physics extraction are much more developed in areas of science
of STEM as a quantitative structural, rather than imaging, tool in such as astronomy, where experimental studies are impossible and,
which the position (and shapes) of atomic columns can be deter- hence, scientific inquiry relies on purely observational data to
mined with ~pm level precision [121]. These developments have establish an underpinning theory.
given rise to significant progress in the visualization of physical In this section, we describe several applications of big data and
functionalities and the mapping of order-parameter fields machine learning tools to explore structural and multispectral data.
including, for example, polarization [122e127] and octahedral tilts In particular, we highlight several functional recognition imaging
in ferroelectrics [124,128e132] and chemical strains in alloys studies, provide examples of deep learning in materials data anal-
[133,134]. Similar advances have occurred in the fields of scanning- ysis and present illustrations of physics extraction in both discrete
probe microscopy, including scanning-tunneling microscopy and continuous systems.
[135,136], contact atomic-force microscopy in liquids and non-
contact atomic-force microscopy (AFM). Equally impressive has 3.1. Order-parameter mapping and local crystallography
been the growth of hyperspectral imaging, in which spectra rep-
resenting materials functionality at a given location is collected as a As noted above, the proliferation of high-resolution STEM im-
function of corresponding stimuli. Examples of this capability aging in the early 2000s has made the imaging of material struc-
include techniques such as electron energy-loss spectroscopy tures with atomic resolution routine. More importantly, this
(EELS) in STEM [137], force-distance curve mapping in AFM, I-V approach has enabled localization of atomic nuclei within columns
curve mapping in STM and hyperspectral imaging in optical and having ~ pm level precision, providing direct information on
mass-spectrometric measurements. symmetry-breaking distortions in solids. This approach was first
As noted in Sec. II.2, this emergence of high-resolution and high- demonstrated by Jia [122,123] for TEM and Chisholm [138], Bor-
veracity structural and hyperspectral data presents many chal- isevich [124] and Nelson [127] for STEM mapping of the polariza-
lenges and opportunities in data storage, analysis and physics tion order parameter of ferroelectrics, providing for the first time
extraction. The scope of these data-related challenges can be sim- real-space imaging of polarization fields with atomic resolution.
ply illustrated by considering the typical size of an electron Several years later, STEM imaging was employed to map oxygen
microscopy-based dynamic movie (or hyperspectral data set), octahedral tilts and their evolution across domain boundaries and
where a modest 1000 frame movie of a dynamic transformation other interfaces, thereby providing a real-space view of the
acquired at 2048  2048 pixel resolution corresponds to a ~18 GB symmetry-breaking physics in these materials [128].
 J.M. Rickman et al. / Acta Materialia 168 (2019) 473e510 489

In the majority of these analyses, the atomic coordinates combined methodologies that directly highlight local contrast,
determined from the STEM image were used as a proxy for the thereby obviating atomic position extraction. This analysis is
corresponding order-parameter field. In the top-down approach of accomplished by first splitting an image into a set of sub-images,
classical physics, the order parameter is defined as an emergent with the sub-image characteristic size larger than the structure of
symmetry breaking distortion across a phase transition. This interest (e.g., a lattice parameter), but well below the image size.
distortion mode is directly related to the atomic displacement These sub-images are then Fourier transformed, generating a 4D set
pattern, and the associated energetics is described by the free en- of diffraction patterns. The linear unmixing of the resultant 4D data
ergy of the system via the Ginzburg-Landau expansion. Corre- sets generates clearly distinguishable abundance maps indicating
spondingly, mapping atomic coordinates permits the estimation of the presence of dissimilar phases, and endmembers defining their
an order parameter. The natural limitation of this mapping method structure, as shown in Fig. 12. These unmixing methods can be
is, however, that STEM provides information only on the co- adapted to allow for the incorporation of physical priors that reflect
ordinates in the image plane, and sometimes only on subset of the known information about the system (e.g., normalized,
atomic species (e.g., only cations in ABO3 perovskites). This spatially sparse abundance maps with non-negative entries that
incomplete information is often used as a proxy for the order have symmetry-constrained endmembers).
parameter field, (e.g., via direct determination of dipole moments
from coordinates and (postulated) Bader or Born charges for po-
3.2. Analysis of multispectral data
larization). Similarly, for octahedral tilts in ferroelectrics, oxygen
atomic positions provide information on the tilt component in the
Multiple imaging modalities give rise to hyperspectral data sets,
image plane. For an analysis along the beam direction, Borisevich
in which the spectrum is generated at each spatial pixel (or voxel).
et al. [124,129] have shown that the column shape can contain
A selection of these modalities would include: 1.) optical and
information on symmetry-breaking distortions.
micro-Raman imaging, 2.) techniques such as ToF SIMS, 3.) EELS in
Very often, the analysis of the data in functional materials is
STEM, 4.) band excitation SPM, and 5.) multiple spectroscopic
complicated by the fact that multiple phenomena can manifest,
modes in scanning-probe microscopies. These imaging modes yield
resulting in similar structural changes. For example, chemical
high-dimensional data sets, as exemplified by the 3D data cubes
concentration gradients and flexoelectric strains are indistin-
common in ToF SIMS, micro-Raman, and EELS. In addition, in
guishable in mesoscopic measurements. Thus, one confronts the
certain SPM methods, the combination of time- and voltage-
challenge of separating dissimilar contributions to the observed
dependent measurements gives rise to 4, 5 and 6D data sets
atomic positions, necessitating complementary imaging modes to
[143e147].
reconstruct the complex local physics and electrochemistry of the
This abundance of data necessitates methods for visualization,
topological defects, surfaces and interfaces. For example, observa-
exploratory analysis, and physics-based analysis of the resultant
tions of the polarization profile, lattice expansion, and the oxygen
data sets. Visualization and exploratory analysis generally aim to
K-edge intensity in EELS evince the nature of screening phenomena
provide a (comprehensible) input to a human operator whose
at ferroelectric-non ferroelectric interfaces, where screening was
judgement is based on expertise in a specific domain field. Classical
found to be induced by oxygen vacancies rather than holes in the
methods for visualization include, for example, exploration of
material [133]. This surprising behavior was observed even at
cross-sections (e.g., image at a given frequency or wavelength);
metal-metal oxide interfaces [139] and attributed to the difference
however, this representation often enhances noise and thereby
in the electrochemical potential of oxygen between constituent
makes detection of small changes spread over multiple spectral
components. Recently, vacancy trapping was observed at the
bins impossible. Another classical approach to data interrogation is
charged domain walls in the ferroelectric tunneling barriers [140]
an analysis based on a priori known (or postulated) physical
and directly linked to the macroscopic transport behavior in these
models, such as functional fits or integration of areas under
devices.
selected regions, with subsequent visualization of derived low-
An alternative approach for the analysis of atomically resolved
dimensional data sets as 2D maps of the fitting parameters or
data is based on local crystallography [141,142]. In this case, the
peak intensities. In this case, the analysis is often compromised by
local structure is examined using statistical methods applied to the
observer bias, and by artifacts due to presence of multiple meta-
relevant structural descriptors, such as the distances between the
stable minima for least-square fits, etc.
atom of interest and the nearest neighbors (i.e., the local chemical
An alternative, data analytical approach for the exploration and
environment). One such descriptor is based on k-means (or any
analysis of hyperspectral data sets is based on multivariate statis-
other) clustering, which differentiates atoms based on their envi-
tical methods such as multiple regression, PCA, non-negative ma-
ronment and roughly describes local chemistry and single-phase
trix factorization (NMF), and Bayesian linear unmixing (BLU). In
regions. This approach is illustrated in Fig. 10, showing a local
these methods, the hyperspectral data set is represented as a linear
analysis of a multi-phase oxide in which regions having dissimilar
combination of known (regression) or unknown endmembers with
chemical bonding are evident.
position-dependent weighting coefficients. A 3D data set is thereby
For spatial regions having identical bonding patterns, a similar
converted into a set of abundance maps representing the spatial
procedure can be employed to highlight local, symmetry-breaking
variation of a certain response and associated endmembers that
distortions (e.g., via a principal-component analysis (PCA) of atomic
embody the response as a function of parameters (e.g., frequency).
coordinates). The results of applying this approach to interrogate
Mathematically, this can be represented as
single M2-phase Mo-V-M-O (M ¼ Nb, Ta, Te and/or Sb) are illus-
trated in Fig. 11, which clearly shows an anti-phase boundary in the
X
k
material and the dominant structural distortion modes. It should be Sðx; RÞ ¼ ai ðxÞwi ðRÞ; (2)
noted that this analysis does not impose an a priori postulated form i¼1
for the order parameter; rather, the aim is to explore universalities
in existing physical distortions. Moreover, by the nature of the where S is the full data set, x is the spatial coordinate, x ¼ (x,y), R is
process, this approach works only if multiple domain variants are the parameter vector, wi ðRÞ are the individual component spectra
present, permitting a statistical determination of existing patterns. (endmembers), and ai ðxÞ are corresponding spatial maps (also
Finally, we note that real-space images can be analyzed by using called abundance maps) representing the spatial localization of
490 J.M. Rickman et al. / Acta Materialia 168 (2019) 473e510

Fig. 10. A two-phase oxide, Mo-V-M-O (M ¼ Nb, Ta, Te and/or Sb). (a) The M1 and M2 mixed phase STEM image. (b) k-means clustering results for 6 neighbors, sorted by distance
metric. (c) k-means clustering results for 6 neighbors, sorted by angle metric. (d) The fast Fourier transform (FFT) of the image in (a). (e) 50-member neighborhood of the image in
(a). (f) 6-member neighborhood of the image in (a). (g) Dendrogram for the 6 neighbors, sorted by distance metric, with the y-axis signifying the cluster separation in the hier-
archical tree. Individual k-means clusters. (h) Cluster 1 spatial distribution with (I) FFT of the distribution and a 2D histogram of neighbors of atoms in the cluster. (i) Cluster 2 spatial
distribution with (I) FFT of the distribution and a 2D histogram of neighbors of atoms in the cluster. (j) Cluster 3 spatial distribution with (I) FFT of the distribution and a 2D
histogram of neighbors of atoms in the cluster. (k) Cluster 4 spatial distribution with (I) FFT of the distribution and a 2D histogram of neighbors of atoms in the cluster (reproduced
from Ref. [141]).
 J.M. Rickman et al. / Acta Materialia 168 (2019) 473e510 491

Fig. 11. Single M2 phase Mo-V-M-O (M ¼ Nb, Ta, Te and/or Sb). (a) M2 phase STEM image. (b) k-means clustering results for 6 neighbor, sorted by distance metric. (c) k-means
clustering results for 6 neighbor, sorted by angle metric. (d) FFT of image in (a). (e) 50-member neighborhood of the image in (a). (f) 6-member neighborhood of the image in (a). (g)
Dendrogram for the 6 neighbor, sorted by angles metric, with the y-axis signifying the cluster separation in the hierarchical tree. PCA loadings and vectors. (h) First eigenvector, in
the upper left, and a corresponding loading map. (i) Second eigenvector, in the upper left, and a corresponding loading map. (j) Third eigenvector, in the upper left, and a cor-
responding loading map. (k) Fourth eigenvector, in the upper left, and a corresponding loading map. (l) Fifth eigenvector, in the upper left, and a corresponding loading map. (m)
Sixth eigenvector, in the upper left, and a corresponding loading map (reproduced from Ref. [141]).

responses described by the endmembers. The functional form materials properties in a complex, non-linear manner. In this latter
given in Eq. (2) clearly constrains the data analysis. In some optical case, general linear unmixing methods work poorly (unless the
methods, for example, the detected signal is a linear combination of response variation is small) [148].
the contributions of individual components whereas, in techniques A second important consideration in the choice and application
such as band excitation SPM, the measured signal depends on local of the aforementioned multivariate methods is the physical
492 J.M. Rickman et al. / Acta Materialia 168 (2019) 473e510

Fig. 12. A real STEM image of a catalyst. (a) The STEM image, with the window from the Sliding FFT overlaid in white. The scale bar is 10 nm. Inset, Global FFT. Results of endmember
extraction with N-FINDR (nc ¼ 3) after Sliding FFT, for the atomically resolved image in (a). (bed) The three endmembers, and (eeg) their spatial abundances are shown. Clearly both
phases are isolated by the algorithm. The width of the window is 500px, and the window step size is 100px (reproduced from Ref. [141]).

constraints on the endmembers and abundance maps. For example, relationships in each system. Ziatdinov et al. [153] employed a
one of the early criticisms of the application of PCA to EELS and combination of sliding window 2D FFT with the Pearson correlation
optical data was that, by the physical nature of the signal, the matrix, global and local Moran's correlative analysis and linear and
detected response is always non-negative. At the same time, the kernel canonical correlation analysis to study a relationship be-
second (and potentially higher-order) principal components will tween structure and electronic parameters extracted from different
necessarily have negative, and therefore unphysical, contributions. “channels” of atomically-resolved scanning-probe microscopy
Therefore, of interest in this context are techniques such as NMF, in measurements on graphene with different types of point defects
which endmembers are non-negative, and BLU, in which the (Fig. 13). The authors found a link between the fine structure of
response is non-negative and the loadings sum to unity. More electron interference patterns and the variations in lattice strain on
complex versions of linear unmixing permit the imposition of the atomic level. In addition, Ziatdinov et al. [154] applied Moran's
constraints on the abundance maps, such as sparsity (small number correlative analysis to discuss the interplay between orientational
of non-zero components) or spatial localization. In short, unmixing and structural degrees of freedom of sumanene molecules from
methods can be tailored to match the physics of the imaging pro- experimental scanning-tunneling microscopy data on sumanene
cess and the material, allowing for the generation of physically self-assembly on metallic surfaces.
significant components [149].
Recent advances in scanning-probe microscopy and scanning 3.3. Functional recognition imaging and physics extraction using
transmission-electron microscopy have opened unprecedented neural networks
opportunities for probing materials structural parameters and
functional properties in real space on both the nanometer scale and Beyond exploratory data analysis and unmixing, hyperspectral
atomic scales. However, until very recently, there were no physics- imaging may be exploited to accomplish more complex analytical
based approaches for rapid data mining of the large acquired data tasks, such as the identification of individual components or a
sets and for cross-correlation of experimental measurements via quantitative analysis of underlying physical phenomena based on
different information channels. To address the first challenge, local responses. Consider, for example, an SPM image of the
Vasudevan et al. demonstrated that combining a sliding-window mixture of two bacteria types, M. Lysodeikticus and P. Fluorescens, on
2D fast Fourier transform (2D FFT) with PCA [150] and geometric a substrate imaged via band-excitation piezoresponse force mi-
endmember extraction [141] permits the automatic mapping of croscopy in a liquid environment, as shown in Fig. 14 [155]. The
different structural phases in scanning-probe and transmission- individual bacteria can be readily identified by their characteristic
electron microscopy images of different functional materials. In shape on the topographic image. At the same time, dissimilar
addition, Strelkov et al. [151] utilized a BLU technique to extract bacteria have different electromechanical behavior, as can be seen
physically meaningful and well-defined spectroscopic components, from the frequency dependence of the electromechanical response.
together with their associated spatial abundance maps, for a series The image formation mechanism in this case is sufficiently complex
of current-voltage measurements on a CoFe2O4-BiFeO3 nano- that the functional form of the response is unknown, thereby
composite thin film. These results provided insight into the spatial precluding quantitative analysis. However, the difference in
distribution of physico-chemical behavior in this system on the response between different bacteria and the substrate can be used
nanometer length scale. Similarly, Ziatdinov et al. [152] used this for unambiguous identification.
Bayesian technique to study the strongly inhomogeneous elec- To enable this recognition imaging, a subset of the images was
tronic response in scanning-tunneling spectroscopy experiments selected as a training set such that responses corresponding to
on the surface of BaFe2As2 in the spin density wave regime. individual bacterial classes may be identified. The dimensionality of
In addition to analyzing structural and functional parameters for the data set is reduced using a PCA, yielding a set of component
each material independently, it is possible to perform a correlative maps with a clearly visible contrast. However, a simple comparison
analysis of the extracted descriptors to establish structure-property of the resulting histograms suggests that a single PCA component
 J.M. Rickman et al. / Acta Materialia 168 (2019) 473e510 493

Fig. 13. A correlative analysis of a structure-property relationship in graphene. (a) Atomically resolved scanning-probe microscopy image of graphene with point defects. (b)
Associated 2D FFT data featuring peaks corresponding to the lattice and Dirac (electronic) structure. (cef) The results of a canonical correlation analysis and Pearson matrix
correlation analysis for graphene samples predominantly populated by point defects passivated with O groups (c, e) and by point defects passivated with H (d, f) (reproduced from
Ref. [153]).

Fig. 14. (a) Surface topography of the M. Lysodeikticus and P. Fluorescens bacteria deposited on a mica surface. (b) The frequency-dependent electromechanical response of individual
bacteria and the substrate in aqueous solution. (c,d) A principal component analysis (PCA) of the response and corresponding loading maps. Note that the relatively broad dis-
tribution of responses does not permit identification based on single pixel data; rather, identification is based only on the PCA component. (e) Neural network based identification of
bacterial types. Shown here is a subset used for training and identification across the image (reproduced from Ref. [155]).

map is insufficient for unambiguous identification. To perform Note that in Fig. 14 a single region on the surface is identified as one
bacterial recognition, the PCA components are used to train a of the bacterial types, presumably due to the presence of a segment
simple back-propagation network employing the PCA components of bacterial shell. At the same time, some of the topographic fea-
as an input and the bacterial type as an output. Thus, the trained tures are identified as a substrate material (or contaminates).
network can identify the bacteria in the remainder of the image An approach similar to that outlined above can be used for more
based on pixel-by-pixel identification (rather than bacterial shape). complex tasks, including the extraction of underlying physical
494 J.M. Rickman et al. / Acta Materialia 168 (2019) 473e510

mechanisms using a network trained on theoretical data to char- below.

acterize the experiment. For example, hysteresis measurements in
piezoresponse force microscopy yield very rich data sets that 3.4. Deep learning for defect and chemical identification
characterize local ferroelectric properties. In particular, each hys-
teresis loop embodies information on the nucleation and growth One of the key tasks in the analysis of a large number of images
mechanisms that characterize the ferroelectric domains below the containing static and dynamic electron and scanning-probe mi-
tip. However, these mechanisms may be deduced by straightfor- croscopy data is rapid feature identification. This capability is
ward numerical analysis only for idealized systems. required for the examination of atomic positions and trajectories
Alternatively, some authors have proposed that hysteretic (on the atomic level) or to establish constituent phase identity and
phenomena can be described by lattice models (e.g., Ising, etc.). In morphology (on the mesoscale), thereby enabling real-time feed-
this regard, Ovchinnikov et al. suggested that a numerically back for global (e.g., pressure, temperature, gas flow) or local
generated set of the hysteresis loops may be used to train a neural (electron beam motion) process control. Such local process control
network in which the PCA components describing the hysteresis is particularly relevant in the context of, for example, e-beam
loop are input parameters and Ising model parameters are outputs atomic fabrication.
(see Fig. 15) [156]. The trained network was shown to be effective in Ziatdinov et al. have recently demonstrated an approach to
parsing the numerical data, and it was then applied to the analysis feature identification that permits deep neural networks trained on
of experimental data [143]. theoretical STM and STEM data to find various atomic [157,158] and
Despite the successes highlighted above, several words of molecular [154] structures in experimental images and to incorpo-
caution are warranted. The use of complex tools, such as neural rate increasingly complex atomic defect configurations into the
nets, for the solution of inverse problems can easily lead to neural network training and classification scheme [157]. Similarly,
improper and incorrect conclusions. For example, the applicability Madsen and co-workers [159] have demonstrated a successful
of the Ising model to ferroelectric switching phenomena is unclear. simulation-to-experiment knowledge transfer for the analysis of
Moreover, the extraction of numerical parameters necessitates a high-resolution TEM data. This analysis of atomically resolved data
very careful examination of uncertainties (“error bars”) and a with deep neural networks may not be limited to real space.
judicious choice of weighting factors for dissimilar PCA compo- Indeed, Vasudevan et al. trained a deep learning model for sym-
nents. Finally, the structure of the neural network is typically metry identification in STEM images using simulated reciprocal
chosen in an ad hoc manner by the operator, and different networks space images having different symmetries.
can lead to disparate results. Many of these issues can be addressed Maksov and co-workers also demonstrated the possibility of
with the incorporation of deep learning in the analysis, as discussed training a deep convolutional neural network using a limited

Fig. 15. (a) Schematics of PCA-NN recognition analysis. (b) Hysteresis loops for a random bond model with dh ¼ 0, for J0 ¼ 2 and dJ ¼ 1, 2, 3, 4, 5. (c) A reconstructed hysteresis loop
using the first 6 eigenvalues for ðJ0 ; dJÞ ¼ (1,1) (red), (1,5,3) (green), and (3,5) (blue). Shown are the original hysteresis loop (solid) and the PCA reconstruction (dashed) (reproduced
from Ref. [156]). (For interpretation of the references to colour in this figure legend, the reader is referred to the Web version of this article.)
 J.M. Rickman et al. / Acta Materialia 168 (2019) 473e510 495

amount of weakly labeled experimental data (see Fig. 16) [160]. ordering and oxygen vacancy behavior at surfaces and interfaces.
More specifically, the authors studied the temporal evolution of Nevertheless, of principal interest here are the relevant mesoscopic
atomic defects in WS2 under e-beam irradiation without employing material parameters (e.g. those needed for a Ginzburg-Landau free
any prior knowledge of the specific types of defects in the system. energy expansions) that can be extracted from these data. In gen-
To train a deep learning model, the authors exploited the fact that eral, the atomic configurations in a material are linked to its ther-
each defect is associated with a violation of ideal periodicity of the modynamics properties, the nature of symmetry-lowering
lattice. Their model was trained using a single image at the early distortions and the broad spectra of interactions that define its
stage of a beam-induced transformation, when macroscopic peri- materials chemistry and physics. Thus, it is of interest to determine
odicity is still maintained, and each defect can be readily discov- whether parameters describing the low-dimensional physics of
ered, providing the “ground truth” for network training. The defect materials (e.g., force fields) can be extracted from images contain-
structures extracted from this trained neural network were then ing information on multiple, spatially resolved degrees of freedom.
classified using unsupervised clustering and unmixing techniques. Recent advances in image analysis indicate that such information is
Fig. 17(a) shows a spatio-temporal diagram of defect trajectories for indeed accessible, and we describe below several investigations of
different classes of defects. From this information Maksov et al. physics extraction in mesoscopic and discrete systems.
were then able to isolate individual ‘flows’ in the 3-dimensional
diagram and analyze the transition probabilities associated with
3.5.1. Reaction-diffusion phenomena
switching between different defect configurations in terms of a
Markov process (Fig. 17(b)). STEM has proven to be a powerful tool for investigating nano-
particle dynamics and growth in chemical, fluidic, and electro-
Very recently, Ziatdinov et al. also combined active electron-
beam manipulation of silicon impurities in graphene during a chemical processes. It permits in-situ investigations of particle
formation and growth with nanometric, and even atomic, spatial
STEM experiment with deep neural network-based image recog-
resolution, and the resulting information can be used to interpret
nition to construct libraries of Si defects in graphene. In addition,
the chemical and physical processes that dictate particle growth
the authors performed STM experiments on the same sample,
and properties (see Fig. 18) [161e167]. However, the analysis of in-
supplemented by first-principles calculations, that allowed them to
situ growth data is hampered by data multidimensionality and size,
link together experimental results that characterize a material's
and therefore requires extensive use of data analytical techniques
structure and functionalities (Fig. 17(c)).
for particle identification and tracking [168e170]. Such algorithms
permit the reduction of data complexity and dimensionality such
3.5. Physics extraction that every particle is characterized by a small number of descriptors
(e.g., center-of-mass position, area and shape).
One of the main goals of image analysis is the extraction of the Particle dynamics information may also elucidate the physical
underlying physical interactions and mechanisms, and the deter- phenomena controlling nucleation and growth. For example, in a
mination of corresponding numerical parameters. For example, recent study based on in-situ particle growth and interaction,
observations of particle evolution in the electrochemical AFM and experimentally measured local particle dynamics is correlated with
liquid-phase electron microscopy experiments provide spectacular simulation [171]. More specifically, a set of reaction-diffusion
snapshots of nucleation and growth processes, and often highlight equations is solved using experimentally observed particle geom-
beautiful associated particle morphologies. For materials applica- etries for different values of the reaction rate, and the corre-
tions, however, one also wishes to determine important governing sponding velocity field is matched with observation to extract the
parameters, such as electrochemical reaction rates and transport associated reaction kinetics. This approach enables the solution of
coefficients. Similarly, over the last decade, STEM has become the the inverse problem for particle growth and permits the identifi-
technique of choice to provide striking images of ordering in ferroic cation of the main control parameters, such as the reaction rate and
materials, ferroelectric and non-ferroelectric phases, octahedral diffusion constant.

Fig. 16. A schematic highlighting the combination of a deep convolutional neural network and a theory-informed Markov model used to create a framework for classification of
states of matter (reproduced from Ref. [160]).
496 J.M. Rickman et al. / Acta Materialia 168 (2019) 473e510

Fig. 17. (a) Spatio-temporal trajectories of atomic defects in WS2 extracted via deep learning analysis of dynamic STEM data. (b) Markov analysis of transition probabilities for
selected (isolated) defect trajectories. (c) Schematics of the creation and exploration of a library of atomic defects (for illustrative purposes, a workflow is shown for a single defect
only) (reproduced from Ref. [160]).

Fig. 18. (a) STEM imaging of platinum nanoparticle growth from aqueous solution; (b) The dynamics of the single particle growth; (c) A simulated map of the platinum con-
centration in the vicinity of growing particles in the static diffusional approximation; (d) A plot of the particle boundary growth velocity (dots e experimental result, line e
simulated data) [11].

3.5.2. Physics of ferroelectrics and vacancy-ordered systems energy functional of a coarse-grained order parameter. The func-
For multiple materials classes, including ferroelectrics and fer- tional form of this free energy is often known from macroscopic
roelastics, the physics of interest can be described in terms of a free- symmetries (scattering data), whereas the corresponding
 J.M. Rickman et al. / Acta Materialia 168 (2019) 473e510 497

expansion coefficients can be deduced from macroscopic mea- the system loses stability and transforms into a modulated phase,
surements (heat capacity, lattice parameter, etc.). However, certain as was previously reported for materials in the vicinity of phase
parameters, including coefficients in the gradient and flexoelectric boundaries [185,186]. Li et al. have demonstrated that computer
terms, and especially boundary conditions defining order param- vision algorithms can be employed for the analysis of these vortex
eter behavior at surfaces, interfaces and other defects, are generally structures, and that subsequent matching to the phase-field model
unknown, with the result that there are severe limits to the prac- permits the estimation of the flexoelectric tensor coefficients, as
tical applications of this class of models. Nevertheless, for cases in shown in Fig. 19.
which a small number of well-defined defects is present, the rele- The feasibility of this coarse-grained approach has also been
vant physical behavior can be extracted using a top-down approach demonstrated for the case of systems characterized by vacancy
in which the analytically (or numerically) determined phase field ordering, as shown in Fig. 20 [187]. In this application, a two-
for a given system geometry and free-energy functional (e.g. known component order parameter is developed, with the scalar compo-
from macroscopic measurements) is fitted to experimental data, nent describing the concentration of oxygen vacancies and the
with the coupling, flexoelectric, gradient terms and boundary vector component describing the ordering. Given the postulated
conditions regarded as fitting parameters. form of the order parameter, the analytical solutions for order
As an example of this approach, Fig. 19 displays a ferroelectric parameter profiles across the antiphase boundaries and normal to
vortex structure. These topological structures in low-dimensional the interface were derived. These solutions depend sensitively on
ferroelectrics were predicted by Naumov [172] and extensively the corresponding gradient terms and boundary conditions at an
explored by the Bellaiche group [172e174], and subsequent STEM interface, and these parameters can be determined from fitting the
studies have led to several examples of closure and vortex domain model to the experimental data. With this information in hand, the
observations, both with STEM [127,175] and piezo-response mi- full order-parameter field near specific defects may be recon-
croscopy [176,177]. In addition, spectacular images of vortices or- structed, as well as the corresponding conjugate fields (such as
dered in dense arrays were obtained by Yadav [178]. With regard to strain components) that are not directly observable.
the theoretical description, it has been found that the structure of
the topological defects, such as domain walls [179e183] and
vortices, is very sensitive to the form of free energy, in particular to 3.5.3. Discrete systems
the gradient and flexoelectric coefficients [184]. The latter describe The extraction of physical properties from image data for
the coupling between the polarization and strain gradients, which discrete systems is similarly challenging. For example, the Ising
are important on the nanoscale but expected to be weak for model can serve as a universal description for ordering phenomena
macroscale systems given the small associated spatial gradients. in materials such as solid solutions, magnets and ferroelectrics. The
Li et al. [125] have determined that the flexoelectric coefficient key parameters of the model describe the local behavior of the
strongly affects the shape of ferroelectric vortices. For example, spins and their interaction with global external fields and neigh-
without flexoelectric coupling, the vortices have a square-like bors. It is then natural to ask whether these parameters may be
shape whereas, for moderate flexoelectric values, they develop extracted from experimental data. The traditional approach to this
asperities. For very high values of the coupling, the ground state of extraction includes the matching of macroscopic behavior (e.g.,
heat capacities, magnetic susceptibilities) described by the model

Fig. 19. (a) High resolution STEM image of the lead titanate-strontium titanate superlattice, and (b) the polarization components extracted from the atomic positions, along with (c)
a vorticity map. (d) The average experimental vortex shape and corresponding shape in phase-field theory for different values of the flexoelectric interactions. (e) A fitting surface
describing agreement between the experiment and theory as a function of the flexoelectric interaction. This can be used to determine the corresponding numerical values (and
uncertainties). (f) A comparison of relevant profiles (reproduced from Ref. [125]).
498 J.M. Rickman et al. / Acta Materialia 168 (2019) 473e510

Fig. 20. Vacancy ordering in lanthanumdstrontium cobaltite deposited on neodymium gallate. Shown here is (a) an atomically-resolved STEM image of the brownmillerite
structure, and (b) the corresponding lattice parameter map, clearly showing an ordering pattern, antiphase boundary, and suppression of the ordering order parameter at the
interface and especially at the AFB-interface junction. (c,d) The order parameter and vacancy concentration maps calculated for the free energy with the gradient and interface
terms extracted from the experimental data. (e,f) The strain components calculated as conjugate fields (reproduced from Ref. [187]).

to the corresponding global experimental measurements. However, machine learning, beginning with relatively established methods
an examination of model behavior for different interaction and then outlining emerging techniques. In applying these strate-
strengths clearly illustrates that the corresponding distribution of gies, however, there are some attendant complications. For
(stochastic) microscopic degrees of freedom are determined by example, in many cases, the nature of the macroscopic order
interaction parameters. This observation suggests that one can parameter is either unknown, its determination hindered by the
match the model to experimental data by using the properly presence of many strongly interacting defects or its existence is
defined statistical properties of the observed images. This approach questionable, as is the case for systems with hierarchical orderings,
was developed by Vlcek et al. [188e190] using the concept of sta- competing ground states or correlated disorder. While considerable
tistical distance minimization for broad classes of experimental information on local degrees of freedom is available from tech-
data, and applied to analyze atomically-resolved images in STM and niques such as (S)TEM, STM, and NC-AFM, one must still determine
STEM, as shown in Fig. 21. whether machine learning can provide the necessary tools to
More generally, access to both averages and atomic-scale fluc- extract physically relevant information.
tuations permits one to deduce local physical functionalities in a In the theory domain, the feasibility of physics extraction was
finite region of the temperature-composition phase diagram. We demonstrated about two years ago (e.g., in a set of publications by
note that such temporal fluctuations are extensively studied in R. Melko and Carrasquila), providing the impetus for many other
fields such as biomolecular unfolding via atomic force microscopy studies. For example, artificial neural networks (ANNs) have been
[191,192] or, more recently, used to obtain equilibrium free energies applied to the detection of phases and phase transitions in classical
from the dynamic trajectories of DNA molecules in a nanofluidic and quantum systems as was recently demonstrated for the 2D
environment [193,194]. Similarly, compositional and structural Ising [195,196] and Potts [158] models, the 3D Hubbard-Fermi
fluctuations in a quenched (static) system can be used to build a model [197], lattice gauge theories [198], Chern insulators [199]
generative model encoding the effective interactions in the system. and the many-body localization problem [200]. In the case of non-
lattice models, ANNs have also been applied to identify transitions
between various states of polymeric systems [201]. In all of these
4. Future opportunities in atomically-resolved imaging
cases, the ANN was trained on labeled lattice (or polymeric) con-
figurations, essentially reducing the problem of detecting phases
Big data and machine learning strategies seek to obtain physi-
and phase transitions to a supervised classification problem.
cally relevant information from mesoscopic and atomically-
Clearly, the application of this approach to a broad range of mate-
resolved data. These goals require the comprehensive use of data
rials- and physics-related problems will require large volumes of
mining and machine learning techniques to extract materials
labeled image data. However, given that the images of many ma-
physics from quantitative measurements of multiple, spatially
terials can look very similar, more complex methods combining
distributed degrees of freedom. For this purpose, we identified in
both image and semantic components are required.
this section three primary strategies for physics extraction using
 J.M. Rickman et al. / Acta Materialia 168 (2019) 473e510 499

Fig. 21. (a) A schematic representation of the framework for model derivation from imaging data, with statistical distance forming the interface between the experimental data and
their theoretical counterpart. (b) An STM image of the FeSe0.45Te0.55 surface exposing the chalcogen layer. (c) The reconstruction of atomic types and positions from image analysis
covering approximately 50  50 atoms; Se (blue), Te (red). (d) The structure of the bulk FeSexTe1-x crystal lattice. (e) The geometry of model interactions defined by Eq. (1), between
Se/Te sites (black) and the neighboring Te/Se sites within the same layer, w0, and the layers below, w-1 and above, w1 (reproduced from Ref. [188]). (For interpretation of the
references to colour in this figure legend, the reader is referred to the Web version of this article.)

It is also possible to “learn” phase transitions without explicitly We note that the application of machine learning methods is
labeling different configurations, effectively using imaging data as fully viable for cases in which some information on microscopic
inputs. One simple way of achieving this aim relies on purposely degrees of freedom is unavailable, provided that a large number of
mislabeling the data (‘learning by confusion’) [202]. This approach (statistically independent) data sets can be measured. One of us has
was demonstrated for topological phase transitions in the Kitaev recently demonstrated the feasibility of this approach for meso-
chain and for the thermal phase transition in the classical Ising scopic data [210]. In particular, it is applicable to STEM data on 2D
model. It was also very recently extended to detect phase transi- materials based on projections corresponding to an average in the
tions from fully unlabeled data with a pair of discriminative z-direction.
cooperative networks (a “guesser” and a “learner”) [203]. A varia- Finally, it has been found that imaging studies can constrain a
tion on this theme is based on a combination of supervised and generative model's parameters to match observed configurations
unsupervised learning, which permits the location of phase tran- with those generated from the model, and the generative model
sitions via the identification of a local maximum in the learning can be used to predict subsequently the configurations as a function
success function [204]. This hybrid approach has also been applied of, for example, temperature and composition. Indeed, one of the
to the analysis of phase transitions in non-trivial many-body pha- authors has previously demonstrated this approach for STM data
ses, such as superfluids. [188]. More recently, we have shown that the use of machine
Phase transitions and associated order parameters can be also learning (specifically, variational autoencoders) on these generated
learned in a completely unsupervised fashion by using a (deep) configurations provides an effective tool to map any anomalies
variational autoencoder (VAE) [205,206]. The latent variables of expected in the phase diagram, even though only one material of a
VAE store the parameters that hold information about quantifiable specific composition is characterized. The partial character of in-
structures on configurations, thereby providing an information- formation available in STEM (projection) will affect these studies as
theoretic analog of an order parameter. For the Ising model, for well. However, these model matching approaches can be realized
example, the latent parameter learned from unlabeled data corre- both in the intrinsic parameter space of the system (i.e., all N3 de-
spond to magnetization [207]. In addition, a recent work on grees of freedom are known), as well as in any derived subspace
reconstructing the electronic structure of a crystal lattice via a (e.g., N2 of projections). This conversion affects the uncertainty in
graph convolutional neural network [208] trained on density recovered parameters that can be estimated through error back-
functional theory data [209] suggests the possibility of studying propagation, and may also create a null space of “inaccessible
electronic (and magnetic) phenomena that accompany certain physics” information.
phase transitions directly from real-space, atomically-resolved
structural data.
One important consideration in these analyses is the partial loss 5. Microstructural informatics: classification and
of information. For example, STEM is effectively limited to 2D characterization of materials microstructures
projections of 3D atomic sequences, and this limitation may affect
the applicability of machine learning methods. In other words, in Many materials are polycrystalline in nature, comprising a
bulk materials containing ~ N3 atoms, only ~ N2 degrees of freedom multitude of misoriented grains that are delimited by grain
are accessible experimentally. Thus, for scanning-probe micro- boundaries [211]. By comparison with single crystals, a poly-
scopy, the effect of sub-surface layers on the behavior of surface crystalline aggregate responds differently to external fields, such as
atoms is undetermined. Even for the case of layered materials, an applied stress, owing to interactions among neighboring grains
while generally all atomic units are visible, some data points may and the presence of grain boundaries. Moreover, the thermo-
also be missing and information from local tilts may be lost. mechanical properties of polycrystals, including their effective
diffusivity and creep behavior, are often dictated by the structure of
500 J.M. Rickman et al. / Acta Materialia 168 (2019) 473e510

internal interfaces [212e214]. From these considerations, it is

evident that characterizing the frequency distribution of grain
sizes, the topology of the grain-boundary network and other
microstructural descriptors is important in the determination of
materials behavior. While human intervention has traditionally
been employed in microstructural interrogation, more recently the
techniques of computer vision and machine learning have been
applied to highlight important patterns in microstructural data
[215].
The coarsening of the grain structure in polycrystalline materials
that occurs over time at elevated temperatures also dictates the
thermomechanical properties of these materials [216e218]. In
characterizing the temporal evolution of a microstructure, it is
useful to identify broadly two regimes, namely normal and
abnormal grain growth (AGG). In the former case, the distribution Fig. 22. A micrograph showing deformation twinning in a Cu-8 at.% Al polycrystalline
of grain sizes is relatively narrow and the corresponding grain-size “TWIP” alloy at a strain of 0.17 (reproduced from Ref. [236]).
distribution is described by a scaling relationship [219]. By contrast,
in the latter case, anisotropy in grain-boundary mobility leads to
unusually large grains over time. The ability to predict and control average grain size, the twin spacing, etc., and these scales deter-
abnormal grain growth is of significant technological significance mine the strength and ductility of the material. In another illus-
as it is often, though not always, associated with the degradation of, tration of the role of multiple scales, in the case of Ni-based
for example, mechanical properties [220]. More generally, a quan- superalloys that find widespread use as components in the aero-
titative understanding of the role of material (e.g., composition) space industry, their desirable mechanical properties (e.g., high
and processing (e.g., temperature, pressure, time) parameters on strength and toughness) depend on the distance between coherent,
the time evolution of a microstructure, and therefore associated intermetallic g precipitates, the average grain size and the distance
properties, is desirable for materials engineering. between polycrystalline representative volume elements [237]. In
The use of materials informatics to quantify salient micro- some cases, such as that of a 5083 commercial alloy, it is advan-
structural features and to establish structure-processing-property tageous to engineer a system with multiple microstructural length
correlations is still relatively new, but a robust set of tools has scales to achieve a balance between strength and ductility [238].
already been developed for this purpose. This set includes de- Given these complexities, one seeks robust microstructural de-
scriptors employed to characterize stochastic point processes, such scriptors that reveal the inherent length scales in a microstructure.
as n-point correlation functions [221e223] and nearest-neighbor For this purpose, it is useful to identify those descriptors that are
distributions [224], and to compile statistics on multidimensional appropriate for heterogeneous materials. One can begin with the
data, such as principal component analysis (PCA) [225,226], ca- assumption that microstructural features can be interpreted in
nonical correlation analysis (CCA) [227,228], etc. In addition, other terms of an experimental realization of a stochastic process [221] in
metrics borrowed from the computer science literature, including which these features occur with certain probability. From this
clustering (i.e., Thornton, Dunn, and purity) [229,230] indices, viewpoint, it is sensible to quantify microstructural characteristics
along with new visualization frameworks, such as parallel co- in terms of descriptors that reflect the distribution of precipitates,
ordinates [231], have enabled workers to find patterns in complex such as the nearest-neighbor distribution, or path lengths, such as
data sets. The identification of such patterns suggests, in turn, the chord-length probability density [239]. More generally, one can
various ways to classify and interpret microstructural data. define a set of n-point correlation functions that embody spatial
In this section, we will describe recent developments in the use associations at different locations in a material [222,240]. This
of materials informatics to quantify the link between materials formalism was developed in other contexts to describe spatial and/
processing and properties and microstructure. This relatively new or temporal correlations associated with stochastic point processes
application of data science has been called microstructural infor- [224], and has been used in statistical physics to highlight average
matics [232,233], and it leverages the machinery of data analytics structure at the atomic scale [241] and to characterize the evolution
and recent progress in 3-D materials characterization [234,235] to of a first-order phase transformation [223]. For a random variable,
! !
describe quantitatively microstructural geometry and to develop fð r Þ; that is a function of position, r , the associated n-point
new materials for a variety of engineering applications. Our aim is function is given by Ref. [239].
to present here a few important uses of microstructural informatics
! ! ! ! ! !
that illustrate its utility in materials interrogation and design. Cn ð r 1 ; r 2 ; : : :; r n Þ ¼ < fð r 1 Þ fð r 2 Þ : : : fð r n Þ > ; (3)

5.1. Tools for microstructural quantification and reconstruction where, in this context, the angle brackets denote an average over an
ensemble of microstructures. In practice, the first two functions
The first step in a microstructural informatics program is the (n ¼ 1,2) in this hierarchy reflect the most important structural
geometric and topological characterization of the relevant features information.
of the system under consideration. These features may sometimes The low-order Cn are especially useful as they facilitate micro-
be deduced from a grain-size distribution and, if multiple phases structural reconstruction. For example, in early work, Sheehan and
are present, also from the average distance between precipitates. In Torquato [242] demonstrated that one can construct or reconstruct
many cases, however, this analysis is complicated by an intrinsic random heterogeneous materials from a set of specified correlation
hierarchy of structural length scales [233] that is inextricably linked functions. Such reconstruction highlight the information content of
to materials properties. Consider, for example, the deformation a microstructure by specifying the number of correlation functions
twinning resulting in a Cu-8 at.% Al polycrystalline alloy at a strain required for an accurate reconstruction. In addition, Fullwood et al.
of 0.17, as shown in the micrograph in Fig. 22 [236]. Clearly, this [240] have employed two-point correlation functions to the rapid
microstructure comprises multiple length scales determined by the reconstruction of both two- and three-dimensional
 J.M. Rickman et al. / Acta Materialia 168 (2019) 473e510 501

microstructures. They note, however, that this methodology is 5.2. Microstructural classification and property linkage
applicable to cases in which the microstructure is discrete both in
space and microstructural state space. Thus, there are limitations to As noted above, microstructural descriptors are extremely use-
this approach in the general case in which C2 is obtained from ful tools for characterizing systems containing easily identified
ensemble averaging. We note that Hansen et al. [243] have also features, such as grains, precipitates, etc. In some cases, however,
reconstructed polycrystalline microstructures by utilizing two- such features may not be easily recognizable and/or there may be
point correlation functions, in this case by employing an exhaus- ambiguities in classifying specific features, such as voids, second-
tive branching technique in which grain orientations are deduced phase particles, etc. In such cases, one seeks other methodologies
from statistical information via a combinatorial analysis. for microstructural recognition and classification that do not
Some authors have suggested alternative microstructural depend on domain knowledge of significant features. In recent
quantification and classification schemes. For example, Schmitz years, significant progress in this regard has been made by applying
et al. [244] developed a metadata-based scheme for characterizing techniques developed in the computer vision and machine learning
3D microstructures. Their scheme is based on a set of geometric communities.
descriptors that are supplemented with property information that In recent work, DeCost and Holm [254] employed computer
can be updated. More recently, Azimi et al. [245] proposed a deep vision techniques to formulate microstructural descriptors that can
learning method for the classification of low-carbon steel micro- be computed in real-time without human intervention. More spe-
structures that employs pixel-wise segmentation using a con- cifically, they apply a “bag of visual features” representation, in
volutional neural network. They conclude that their approach which there is an associated probability distribution of images, to
provides a robust and effective performance improvement relative highlight the defining features of a microstructure. A support vector
to other competitive methodologies. Gola et al. [246] apply a data machine then uses this information to classify images using
mining process with a SVM classifier to construct a model that can training sets. Building on this work, Chowdhury et al. [255] also
distinguish among microstructures associated with two-phase employed computer vision and machine learning methods to
steels. In their study, different morphological parameters were recognize complex features, namely dendrites in the microstruc-
employed to discriminate successfully among martensitic, pearlitic ture of solder alloys. In their studies, dimensionality reduction,
and bainitic steels. Finally, Velichko and coworkers [247] also feature extraction techniques and feature classification were ach-
employed data mining to classify graphitic morphologies in cast ieved using support vector machines. Additional uses of machine
iron. They used optical micrographs to identify relevant micro- learning techniques to interpret metallurgical microstructures
structural parameters that enabled them to highlight clusters and include the application of artificial neural networks to perform
trends in their data. image registration (i.e., alignment) [256] and the application of the
If a microstructure contains second-phase particles, it is often “random-forest” statistical learning algorithm to perform image
desirable to characterize their shapes and their evolution over time. segmentation and microstructural pattern recognition [257].
While this is often done by visual inspection, a more quantitative Machine learning has also been employed to connect materials
approach is needed for detailed comparisons between, for example, microstructures with physical properties. For example, Liu et al.
images generated by experiment and computer simulation. For this [258] applied these techniques to a materials property optimization
purpose, MacSleyne et al. [248] also examined contributions to the problem. In particular, they examined microstructures associated
two-point correlation function and, following Hu [249], focused on with an iron-gallium alloy and identified a subset of these struc-
! !
the moment invariants of a particle shape function, Dð r Þ. (Dð r Þ is tures having optimal magnetoelastic properties, such as a low
!
defined to be 1 if the position vector, r , is inside a particle and elastic modulus and a high magnetostrictive strain. Moreover, for
0 otherwise.) Their analysis yields a catalog of moment invariants their studies, they found that machine learning outperformed other
for specific particle shapes, and thereby provides a systematic optimization strategies, such as linear programming and genetic
means for comparing and contrasting complex particle shapes. algorithms. In recent work, Kondo et al. [259] applied convolutional
Finally, we note that the analysis of microstructures outlined neural networks (CNN) to link microstructures in ceramics with
above can be generalized to describe the spatio-temporal evolution their associated ionic conductivities. A CNN is ideally suited to
of an evolving phase resulting from a first-order phase trans- image recognition and has the advantage in this context of being
formation. These transformations proceed by the nucleation of a free from a researcher's feature bias. The authors have also
stable, product phase in a metastable background, and the subse- demonstrated that, at least for some problems, one can use rela-
quent growth of this phase until impingement results in a product- tively small training sets to obtain good correspondences between
phase microstructure. In recent years, an n-point correlation anal- structure and properties. Cecen et al. [260] also employ a CNN to
ysis similar to that summarized above has been applied to obtain connect microstructures with (homogenized) properties. In their
the low-order (equal-time) n-point correlation functions that cap- approach, a CNN is used to identify important microstructural
ture the statistical properties of a system of impinging phase feature that are associated with properties of interest and as esti-
droplets. More specifically, the correlation function formalism of mators of n-point correlation functions. In addition, Liu et al. [261]
Sekimoto [250] captures the evolution of nucleation and growth used an artificial neural network to optimize the microstructure
processes in terms of the n-point correlation function and predicted the ultimate tensile strength of Nb-Si alloys. This
analysis suggested a new design strategy for these materials.
Cn ðr 1 ; r2 ; : : :rn ; t Þ ¼ < Pni¼1 uðri ; tÞ > ; (4) Finally, we note that Exl et al. [262] used a machine learning al-
gorithm to identify microstructural characteristics leading to
where uðr i ; tÞ is an indicator function that reflects whether a region magnetization reversal in Nd2Fe14B permanent magnets.
is transformed or not. For n ¼ 1, one recovers the volume-fraction of
the system that is transformed at a given time, a quantity that was 5.3. Quantification and analysis of abnormal grain growth
first examined by Kolmogorov [251] and Johnson, Mehl and Avrami
[252,253]. The two-point function, by contrast, yields information As noted above, the ubiquitous phenomenon of abnormality in a
on the evolving correlations at two different spatial points by microstructure involves the unusually rapid growth of a minority of
reflecting whether these two points have, on average, transformed constituent grains, and the resulting bimodal structure may have a
at the same time. deleterious impact on the thermomechanical properties of a
502 J.M. Rickman et al. / Acta Materialia 168 (2019) 473e510

system [263]. A proper characterization and quantification of AGG directions in a multidimensional space, one for each set, that are
requires, in principle, monitoring the temporal evolution of a grain maximally correlated, with dimensional reduction resulting from
population to identify those grains that are growing faster, on retaining only a subspace of important directions. The result of this
average, than the rest. In practice, however, abnormality is often analysis is a ranked list of correlation coefficients and associated
inferred from a micrograph that represents a microstructural canonical weights, the latter giving the relative contributions of
snapshot at a given time, often using heuristic criteria associated each member to its associated, so-called canonical variate [267].
with grain size. Thus, one often finds that abnormality has been We note that various improvements to CCA have been advanced in
attributed to microstructures containing a single grain that is larger recent years that permit one to explore non-linear correlations
than some arbitrary multiple of the average grain size. among input and output sets of variables [266,268,269].
This situation is unsatisfactory as there is little consensus as to In their study, Lawrence et al. [264] considered 68 specialty
the specific grain size and shape features that characterize AGG and alumina powders having 11 distinct powder chemistries. These
therefore, until recently [264], a robust set of quantitative AGG powders were spark-plasma sintered and subjected to annealing
indicators has also been lacking. This set can be constructed from treatments for a prescribed time, t, and temperature, T. To charac-
the lower-order moments of the joint distribution of grain size, G, terize the resulting microstructures, they compiled statistics on
and area, a, denoted by pðG; aÞ, focusing necessarily on the tails of from 1000 to 9000 grains per sample. To maintain generality, we
the distribution since AGG is, in this context, a rare event. More denote the set of M (input) processing variables by fxi ; i ¼ 1; 2; : : :;
specifically, borrowing from the mathematical framework used to Mg. This set includes, in this case, the compositions of the powder
characterize risk and extreme events in the insurance industry (e.g., dopants, such as MgO, CaO, Na2O and SiO2, along with t and T.
a one-hundred year flood) and in the financial markets (e.g., very Fig. 23 show typical microstructures corresponding to different
large losses) [265], a set of metrics couched in terms of conditional powder chemistries and processing conditions and clearly high-
expectation values may be constructed. These metrics represent, in lights the extreme sensitivity of microstructure to the processing
effect, the shape of the tail of the distribution; that is, they reflect variables. In particular, Fig. 23b and c shows microstructures with
the average grain size, the variance in the grain size, the correlation very different grain morphologies that might be classified as
between grain size and aspect ratio, etc., given that there is an abnormal by different workers. Given these differences, it is useful
extreme event. The set of N metrics will be denoted here by fyi i ¼ to have a representation that captures important microstructural
1; 2; : : :; Ng, and the reader is referred to Ref. [264] for further features. Fig. 24 displays such a representation, namely the corre-
details. sponding microstructural “fingerprints”, and these density maps of
Having obtained a quantitative description of AGG, one can pðG; aÞ are in a format that readily permits a comparison of
employ the methods of materials informatics to explore links be- different grain populations.
tween AGG and materials properties or connections between ma- The CCA for this data M identifies the linear combinations of ca-
P ðkÞ PN
terials processing and the presence of AGG. For brevity, we focus nonical variates V ðkÞ ¼ ai xi and W ðkÞ ¼ bjðkÞ yj that are
here on the latter correlations and, to illustrate the methodology maximally correlated, where i¼1 k ¼ 1; 2; : : : ; minðM;
j¼1 NÞ and the co-
ðkÞ ðkÞ
used to link processing and microstructural variables, consider the efficients ai and bj are the canonical weights. The corresponding
case of ceramic powder processing. ranked correlation coefficients, lðkÞ , are also obtained and measure
the degree of association between the variates. We note that
5.3.1. Example: processing of specialty alumina powders Rickman et al. [266] also performed an improved, Monte Carlo
Lawrence et al. [264] and Rickman et al. [266] examined the based CCA (called CCAMC) for this data in which another set of
interdependence of materials processing and microstructural var- input variables that contained non-linear functions of the xi was
iables for various doped, specialty alumina powders using a ca- used. Fig. 25 show plots of the first variate pairs, V ð1Þ and W ð1Þ
nonical correlation analysis (CCA). CCA is a general and extremely (denoted for simplicity as just V and W), for both a conventional
versatile methodology for quantifying linear correlations between CCA and a CCAMC having values of lð1Þ 0.59 and 0.86, respectively,
two sets of variables (an input set controlled by an engineer and the along with the corresponding regression lines. Clearly, the CCAMC
resulting output set) and is essentially the most general parametric analysis is a significant improvement over the conventional
statistical test of significance [228]. This analysis finds projected analysis.

Fig. 23. The microstructures resulting from the sintering of three specialty alumina powders having different powder chemistries and/or processing conditions. The dopants and
processing conditions are: a.) MgO and hot pressing at 1750  C for 2 h b.) MgO and CaO and spark-plasma sintering at 1300  C and c.) CaO and SiO2 and hot pressing at 1750  C for
2 h (reproduced from Ref. [264]).
 J.M. Rickman et al. / Acta Materialia 168 (2019) 473e510 503

Fig. 24. The “fingerprints” corresponding to the three microstructures shown in Fig. . These quantities are density maps of pðG; aÞ: For convenience, these maps are given in terms of
b and average aspect ratio, ba, respectively (reproduced from Ref. [264]).
the average grain size, G,

Fig. 25. The canonical variates, V ð1Þ and W ð1Þ (denoted by simply V and W) for the a.) conventional CCA and the b.) CCAMC analysis. In each case the points represent the 68 different
powders and the solid line is a regression line. In b.) a 95% confidence interval line is also shown. It should be noted that the CCAMC analysis is a significant improvement over the
conventional analysis (reproduced from Ref. [266]).

Beyond the identification of important combinations of pro- application. Moreover, to provide some context for this relatively
cessing and microstructural variables, the CCA (or CCAMC) provides new field, we have discussed the use of information sciences in
a means for experimental planning. In particular, if one desires a materials design from a historical perspective and presented the
0 0
particular microstructure, as described by fyi g or, equivalently, W , current state of the materials data landscape.
where the prime denotes specific values, one can select a particular Looking to the future, there are many potential opportunities
0 0
set of processing variables, fxi g, that yields the corresponding V and challenges for materials informatics. As noted earlier, the Ma-
from the appropriate regression line. Finally, we note that CCA and terials Genome Initiative (MGI) [32], a multi-agency program
CCAMC are general techniques that can be applied to a wide variety launched in 2011, has provided the impetus to develop new
of problems in which one wishes to quantify the correlations be- computational approaches to accelerate materials discovery and
tween a set of input and a set of output variables. A recent example enable property optimization. One principle of MGI is that high-
explores the correlations between microstructure and the electrical throughput data science will be beneficial in achieving this accel-
and optoelectronic properties of thin-film solar cells [266,270]. eration. While a central aspect of data science involves the creation
and curation of data repositories, MGI researchers today advocate
more broadly for an interactive materials “ecosystem” at the
6. Discussion and outlook intersection of experiment, computation and data science [271].
Moreover, given the geographical distribution of resources such as
In this Overview we have described the contents of the mate- data, equipment and domain expertise, it has also been recognized
rials informatics toolbox and the use of informatics to help analyze that electronic collaboration (e-collaboration) is vital to address
and interpret large, multidimensional datasets that characterize grand-challenge materials science problems [271]. In this regard, it
materials structure and functionality. The resulting analyses can is also worth noting that new, innovative facilities are envisioned to
encompass many disparate materials systems and a wide range of facilitate these collaborations. For example, a unique, interdisci-
thermomechanical, electromagnetic and chemical properties. Our plinary facility, a Nano/Human Interface (NHI), for data-driven
discussion has provided a snapshot of the field highlighting appli- discovery in which humans interact immersively with machines
cations of informatics methods to materials design, microstructural to interpret large, multi-dimensional data sets is a partnership
quantification, model development, characterization of spatial between Lehigh University and the Ohio State University and is
ordering, etc., as well as a guide to the best tools for a given
504 J.M. Rickman et al. / Acta Materialia 168 (2019) 473e510

currently under construction.

The materials of the future must be rapidly fabricated and
designed subject to specific performance requirements. One engi- Acknowledgements
neering design approach to achieve this objective, known as Inte-
grate Computational Materials Engineering (ICME), involves the This work was supported by the U.S. Department of Energy
synergy between modeling and simulations and materials data- (DOE), Office of Science, Materials Sciences and Engineering Divi-
bases, as well as innovations in manufacturing. Its goal is also to sion (S.V.K.). Research was conducted at the Center for Nanophase
reduce the time from development to market for promising ma- Materials Sciences (SVK), which is a DOE Office of Science User
terials by employing multiscale modeling. To be successful, these Facility. One of the authors (SVK) greatly acknowledges help from
models must reflect intrinsic structural hierarchies in materials R.K. Vasudevan, M. Ziatdinov, and A. Ievlev. JMR thanks Lehigh’s
(e.g., as found in polycrystalline materials) [272]. Thus, it is ex- Nano/Human Interface initiative for its support.
pected that materials informatics, with its capabilities for inverse
design and uncertainty quantification, will play an important role Appendix
in advancing the ICME objectives. Finally, we note that one rela-
tively new application of informatics that has the potential to In this Overview we describe the use of several methodologies,
transform materials research is text mining. With text mining tools developed primarily outside the domain of materials science, to the
one can extract data from online publications and use this infor- solution of a wide range of materials problems. It is therefore
mation in some materials context [273]. desirable to present a brief survey of these methodologies,
Data is, of course, central to the whole materials informatics including their definitions and a summary of their utility, so that
enterprise. Given the discussion above (especially Sec. II.3), it is the terminology employed here is clear. The following is, then, an
evident that data management will be an important consideration abbreviated glossary with some useful references.
in ongoing and future collaborations. Data management or curation Artificial Intelligence (AI): The field of computer science that
refers to “on-going management of data through its lifecycle of pertains to the synthesis and subsequent analysis of objects that act
interest and usefulness” that enables subsequent data discovery on their environment (agents) in an intelligent way and whose
[274]. This activity comprises several interrelated tasks including: actions can be described in terms of computational operations
data creation, classification, validation and preservation [275]. The [279].
importance of data curation in a research environment is empha- Bayesian Network: A set of variables and edges comprising a
sized by the National Science Foundation's mandate of a two-page directed, acyclic probabilistic graph that has an associated table of
data management plan for proposals that are submitted to the conditional probabilities [280].
agency. Several institutions, including the National Institute for Canonical Correlation Analysis (CCA): A generic, parametric
Standards and Technology (NIST) and the Research Collaboratory model employed in statistical data analysis in which, for two paired
for Structural Bioinformatics (RCSB), have already begun to address data sets, paired directions (1 and 2) are identified such that the
many of the main issues involved in data management and have projection of the first data set along the direction 1 is maximally
established the requisite infrastructures for this purpose, such as correlated with the projection of the second data set along direc-
the Materials Data Curation System (NIST) and the Protein Data tion 2 [227,228]. CCA is a dimensional reduction technique.
Bank (PDB) [276]. Nevertheless, important issues remain and limit Cluster Analysis: A collection of diverse techniques for identi-
the participation of some researchers and institutions in the pro- fying structures within data sets. In these approaches, data are
cess of data discovery and innovation [277]. We conclude with a grouped such that the elements of a particular group have some
non-exhaustive list of some important data-related issues and degree of association [281].
associated questions. Data Mining: The discovery of interesting and novel patterns,
along with predictive models, from a large volume of data. Some of
 Data ownership and sharing the methods used for this purpose include pattern mining, clus-
tering and classification [282].
How can one promote data sharing given proprietary concerns?
Deep Learning: Neural networks having a large number of pa-
Who is responsible for data maintenance? rameters and layers in a particular architecture (e.g., convolutional
neural network, recurrent neural network). One advantage of deep
 Metadata Curation learning versus more traditional approaches is the ability of the
network to decide which aspects of a dataset indicate data reli-
Metadata is essentially a set of data that describes another set of
ability (i.e., automatic feature extraction). As compared to other
data. It describes a digital resource enabling accurate and effi-
multilayer networks, deep learning networks tend to have more
cient recovery of information and verification of its integrity
neurons and interlayer connectivity is more complex [283].
[278].
Dimensionality Reduction: The summary of a large set of
What are appropriate metadata schema? interrelated features or parameters by a smaller set having less
redundancy that highlights the most important dimensions. This
What are the best methods for metadata preservation and
technique is useful in mitigating the effects of the “curse of
curation?
dimensionality”, including multi-dimensional visualization diffi-
culties and the inherent sparseness of high-dimensional spaces
 Managing Data From Multiple Sources
[284].
How do we best cope with heterogeneous data (e.g., data se- Genetic Algorithm (GA): An algorithmic analogy of the biolog-
quences, graphs, micrographs, patterns) obtained from many ical concept of evolution and, in particular, “survival of the fittest”.
sources? This approach is based on the evolution of candidate solutions to an
optimization problem under the processes of crossover and mu-
How can heterogeneous data be made useful to practitioners in
tation [285].
diverse fields?
Hyperparameter: Parameters that are employed to tune a
network for machine learning to improve the functionality of the
 J.M. Rickman et al. / Acta Materialia 168 (2019) 473e510 505

network. The selection of these parameters is done so that the learning, J. Materiomics 3 (2017) 159e177.
[11] J. Carrete, N. Mingo, S. Wang, S. Curtarolo, Nanograined half-Heusler semi-
associated model does not overfit or underfit a given training set
conductors as advanced thermoelectrics: an ab Initio high-throughput sta-
[283]. tistical study, Adv. Funct. Mater. 24 (2014) 7427e7432.
Machine Learning (ML): The process of programming com- [12] S. Kirklin, B. Meredig, C. Wolverton, High-throughput computational
puters to optimize some (performance) criterion using either screening of new Li-ion battery anode materials, Adv. Energy Mater. 3 (2013)
252e262.
experience or training data. It uses statistical methods to build [13] S. Chakraborty, W. Xie, N. Mathews, M. Sherburne, R. Ahuja, M. Asta,
mathematical models based on some parameter set, and learning S.G. Mhaisalkar, Rational Design: a high-throughput computational
takes place by parameter optimization via computation [286]. screening and experimental validation methodology for lead-free and
emergent hybrid perovskites, ACS Energy Lett. 2 (2017) 837e845.
Neural Network (NN): An interconnected collection of nodes [14] Y. Zhang, C. Ling, A strategy to apply machine learning to small datasets in
whose functionality mimics that of an animal neuron. Node con- materials science, npj Comp. Mat. 4 (2018) 25.
nectivity is specified in terms of a series of weights that are learned [15] P.V. Balachandran, J. Young, T. Lookman, J.M. Rondinelli, Learning from data
to design functional materials without inversion symmetry, Nat. Comm. 8
from a training set [287]. A convolutional neural network (CNN) is a (2017) 14282.
multi-layer neural network that is often applied to images. The [16] M. deJong, W. Chen, R. Notestine, K. Persson, G. Ceder, A. Jain, M. Asta,
architecture is chosen so as to minimize the number of model pa- A. Gamst, A statistical learning framework for materials science: application
to elastic moduli of k-nary inorganic polycrystalline compounds, Sci. Rep.-UK
rameters that must be learned [288]. 6 (2016) 34256.
Perceptron: A binary classifier based on a linear discriminant [17] N. Wagner, J.M. Rondinelli, Theory-guided machine learning in materials
function [289]. A perceptron may be either single- or multi-layered science, Front. Mater. (2016), https://doi.org/10.3389/fmats.2016.00028.
[18] J. Schmidt, J. Shi, P. Borlido, L. Chen, S. Botti, M.A.L. Marques, Predicting the
neural network models that can be trained.
thermodynamic stability of solids combining density-functional theory and
Principal Component Analysis (PCA): A dimensional reduction machine learning, Chem. Mater. 29 (2017) 5090e5103, 12.
technique in which there is a transformation to a new set of or- [19] E. Mooser, W.B. Pearson, On the crystal chemistry of normal valence com-
dered, uncorrelated variables (the principal components) such that pounds, Acta Crystallogr. 12 (1959) 1015.
[20] J. St. John, A.N. Bloch, Quantum defect electronegativity scale for non-
the first few principal components retain much of the variation in transition elements, Phys. Rev. Lett. 33 (1974) 1095.
the original variables [290,291]. [21] J.R. Chelikowsky, J.C. Phillips, Quantum defect theory of heats of formation
Probability: In the frequentist view, the probability of occurrence and structural transition energies of liquid and simple solid metal alloys and
compounds, Phys. Rev. B 17 (1978) 2453.
is dictated by the frequency that a given outcome occurs as a [22] J.C. Phillips, J.A. van Vechten, Dielectric classification of crystal structures,
measurement is made in many blind trials. In short, one focuses on ionization potentials and band structures, Phys. Rev. 22 (1969) 705.
the frequency of occurrence in the long run. By contrast, in the [23] J.C. Phillips, Ionicity of chemical bonds in crystals, Rev. Mod. Phys. 42 (1970)
317.
Bayesian view, probability is linked with beliefs about a given var- [24] A. Zunger, Structural stability of 495 binary compounds, Phys. Rev. Lett. 44
iable, and these beliefs are updated based on new information (1980) 582.
[283]. [25] A. Zunger, Systemization of the stable crystal structure of all AB-type binary
compounds: a pseudopotential orbital-radii approach, Phys. Rev. B 22 (1980)
Statistical Distance: This terms refers to the separation between 5839.
statistical objects such as distribution functions, density functions [26] D.G. Pettifor, A chemical scale for crystal-structure maps, Solid State Com-
and moment-generating functions. Specific examples of common mun. 51 (1984) 31e34.
[27] Y. Saad, D. Gao, T. Ngo, S. Bobbitt, J.R. Chelikowsky, W. Andreoni, Data mining
distance measures include the Euclidean distance (from the Py-
for materials: computational experiments with AB compounds, Phys. Rev. B
thagorean theorem), the chi-square distance and the Mahalanobis 85 (2012) 104104.
distance [292]. [28] L.M. Ghiringhelli, J. Vybiral, S.V. Levchenko, C. Draxl, M. Sche_er, Big data of
Support Vector Machine (SVM): A method for solving pattern materials science - critical role of the descriptor, Phys. Rev. Lett. 114 (2015)
105503.
recognition problems in which data is mapped into a higher- [29] G. Pilania, J.E. Gubernatis, T. Lookman, Structure classification and melting
dimensional space and an optimal separating hyperplane is con- temperature prediction in octet AB solids via machine learning, Phys. Rev. B
structed for classification purposes [293]. 91 (2015) 214302.
[30] G. Pilania., J. E. Gubernatis and T. Lookman, Classification of octet AB-type
Unsupervised Learning: This machine learning task occurs binary compounds using dynamical charges: a materials informatics
when the learner must identify patterns or categories in the data perspective, Sci. Rep. j 5:17504 j DOI: 10.1038/srep17504.
without prior specification of these patterns. By analogy with hu- [31] M.F. Ashby, Materials Selection in Mechanical Design, third ed., Elsevier -
Butterworth-Heinemann, 2005.
man learning, unsupervised learning corresponds to learning [32] Materials Genome Initiative for Global Competitiveness, 2011. https://
without a teacher [225,294]. obamawhitehouse.archives.gov/mgi.
[33] P. Dimitri, Solomatine and Durga Lal Shrestha, A novel method to estimate
model uncertainty using machine learning techniques, Water Resour. Res. 45
References (2009) W00B11, https://doi.org/10.1029/2008WR006839.
[34] Anubhav Jain, Geoffroy Hautier, Charles J. Moore, Shyue Ping Ong, Christo-
[1] O.Y. Al-Jarrah, P.D. Yoo, S. Muhaidat, G.K. Karagiannidis, K. Taha, Efficient pher C. Fischer, Tim Mueller, Kristin A. Persson, Gerbrand Ceder, A high-
machine learning for big data: a review, Big Data Res. 2 (2015) 87e93. throughput infrastructure for density functional theory calculations, Com-
[2] A.H. Alavi, A.H. Gandomi, Big data in civil engineering, Autom. ConStruct. 79 put. Mater. Sci. 50 (2011) 2295e2310.
(2017) 1e2. [35] A. Jain, S. Ong, G. Hautier, W. Chen, W.D. Richards, S. Dacek, S. Cholia,
[3] F.N. Catbas, M. Malekzadeh, A machine learning-based algorithm for pro- D. Gunter, D. Skinner, G. Ceder, K. Persson, Commentary: the materials
cessing massive data collected from the mechanical components of movable project: a materials genome approach to accelerating materials innovation,
bridges, Autom. ConStruct. 72 (2016) 269e278. Apl. Mater. 1 (2013) 011002.
[4] L. Wang, C.A. Alexander, Big data in design and manufacturing”, Am. J. Eng. [36] J.E. Saal, S. Kirklin, M. Aykol, B. Meredig, C. Wolverton, Materials design and
Appl. Sci. 8 (2) (2015) 223. discovery with high-throughput density functional theory: the open quan-
[5] J. Brandenburger, V. Colla, S. Cateni, A. Vignali, F. Ferro, C. Schirm, J. Melcher, tum materials database (OQMD), JOM 65 (2013) 1501e1509.
Applying big data concepts to improve flat steel production processes, in: [37] S. Curtarolo, et al., AFLOWLIB.ORG: a distributed materials properties re-
S.S. Roy, R. Deo, P. Samui, S. Ntalampiras (Eds.), Big Data in Engineering pository from high-throughput ab initio calculations, Comput. Mater. Sci. 58
Applications, Springer Nature, Singapore, 2018. (2012) 227e235.
[6] J. Castagna, Data science can be an attractive career for physicists, Phys. [38] U. Lienert, J. Almer, B. Jakobsen, W. Pantleon, D. Hennesey, C. Xiao,
Today 69 (8) (2016) 20. R.M. Suter, 3-dimensional characterization of polycrystalline bulk materials
[7] G. Bell, in: T. Hey, S. Tansley, K. Tolle (Eds.), The Fourth Paradigm: Data using high-energy synchrotron radiation, Mater. Sci. Forum 539e543 (2007)
Intensive Scientific Discovery, Microsoft Research, Redmond, WA, 2009. 2353e2358.
[8] K. Rajan, Materials informatics, Mater. Today 8 (2005) 38e45. [39] Reeju Pokharel, Jonathan Lind, Shiu Fai Li, Kenesei Peter, Ricardo
[9] B.R. Goldsmith, M. Boley, J. Vreeken, M. Scheffler, L.M. Ghiringhelli, Uncov- A. Lebensohn, Robert M. Suter, Anthony D. Rollett, In-situ observation of bulk
ering structure-property relationships of materials by subgroup discovery, 3d grain evolution during plastic deformation in polycrystalline Cum, Int. J.
New J. Phys. 19 (2017) 013031. Plast. 67 (2015) 217e234.
[10] Y. Liu, T. Zhao, W. Ju, S. Shi, Materials discovery and design using machine [40] S.F. Li, R.M. Suter, Adaptive reconstruction method for three- dimensional
506 J.M. Rickman et al. / Acta Materialia 168 (2019) 473e510

orientation imaging, J. Appl. Cryst. 46 (2013) 512e524. Nat. Rev., Chem. 2 (2018) 0121.
[41] N. Lubbers, T. Lookman, K. Barros, Inferring low-dimensional microstructure [72] F.A.C. Viana, T.W. Simpson, V. Balabanov, V. Toropov, Special section on
representations using convolutional neural networks, Phys. Rev. E 96 (2017) multidisciplinary design optimization: metamodeling in multidisciplinary
052111. design optimization: how far have we really come? AIAA J. 52 (2014)
[42] M. Ramoni, P. Sebastini, Robust learning with missing data, Mach. Learn. 45 670e690. https://doi.org/10.2514/1.J052375.
(2001) 147e170. [73] A. Mannodi-Kanakkithodi, G. Pilania, T.D. Huan, T. Lookman, R. Ramprasad,
[43] Z. Ghahramani, M.I. Jordan, Supervised learning from incomplete data via an Machine learning strategy for accelerated design of polymer dielectrics, Sci.
EM approach, Adv. Neural Inf. Process. Syst. (1994) 120e127. Rep. 6 (2016) 20952. https://doi.org/10.1038/srep20952.
[44] https://nomad-coe.eu. [74] W.B. Powell, I.O. Ryzhov, Optimal Learning, Wiley, 2012.
[45] https://cmr.fysik.dtu.dk. [75] T. Lookman, Frazier, Aggarwal Wang, et al., in: T. Lookman, F.J. Alexander,
[46] http://www.aiida.net. K. Rajan (Eds.), Information Science for Materials Discovery and Design,
[47] N. Mounet, M. Gibertini, P. Schwaller, et al., Two-dimensional materials from Springer International Publishing, 2016.
high-throughput computational exfoliation of experimentally known com- [76] T. Lookman, P. Balachandran, D. Xue, J. Hogden, J. Theiler, Statistical infer-
pounds, Nat. Nanotechnol. 13 (2018) 246e252. ence and adaptive design for materials discovery, Curr. Opin. Solid State
[48] A.K. Singh, K. Mathew, H.L. Zhuang, Richard G. Hennig, Computational Mater. Sci. (2016). https://doi.org/10.1016/j.cossms.2016.10.002.
screening of 2D materials for photocatalysis, J. Phys. Chem. Lett. 6 (2015) [77] M. Cavazzuti, Optimization Methods, Springer-Verlag Berlin Heidelberg,
1087e1098. 2013.
[49] M. Ashton, J. Paul, S.B. Sinnott, R.G. Hennig, Topology-scaling identification of [78] A.I.J. Forrester, A. Sobester, A.J. Keane, Engineering Design via Surrogate
layered solids and stable exfoliated 2D materials, Phys. Rev. Lett. 118 (2017) Modelling: A Practical Guide, John Wiley & Sons, Ltd., 2008.
106101. [79] D.V. Lindley, A.F.M. Smith, Bayes estimates for the linear model, J. Roy. Stat.
[50] S. Haastrup, M. Strange, et al., The computational 2D materials database: Soc. B 34 (1972) 1e41.
high-throughput modeling and discovery of atomically thin crystals, 2D [80] L.A. Dalton, E.R. Dougherty, Optimal classifiers with minimum expected error
Mater. 5 (2018), 042002. within a Bayesian frameworkdPart I: discrete and Gaussian models, Pattern
[51] S. Sanvito, et al., Accelerated discovery of new magnets in the Heusler alloy Recogn. 46 (2013) 1301e1314.
family, Sci. Adv. 3 (2017) e1602241. [81] Xiaoping Du, Wei Chen, Efficient uncertainty analysis methods for multi-
[52] M. Rupp, A. Tkatchenko, K.-R. Müller, O.A. von Lilienfeld, Fast and accurate disciplinary robust design, AIAA J. 40 (2002) 545e552.
modeling of molecular atomization energies with machine learning, Phys. [82] R. Dehghannasiri, D. Xue, P.V. Balachandran, M.R. Youse, L.A. Dalton,
Rev. Lett. 108 (2012) 058301. T. Lookman, E.R. Dougherty, Optimal experimental design for materials
[53] Felix A. Faber, Lindmaa Alexander, O. Anatole von Lilienfeld, discovery, Comp. Mat. Sci. 129 (2017) 311.
Rickard Armiento, Machine learning energies of 2 million elpasolite [83] J. Ling, M. Hutchinson, E. Antono, S. Paradiso, B. Meredig, High-dimensional
(ABC2D6), Phys. Rev. Lett. 117 (2016) 135502. materials and process optimization using data-driven experimental design
[54] P. Gorai Yan, B. Ortiz, S. Miller, S.A. Barnett, T. Mason, V. Stevanovic, with well-calibrated uncertainty estimates, Integrating Mater. Manuf. Innov.
E.S. Toberer, Material descriptors for predicting thermoelectric performance, 6 (2017) 207.
Energy Environ. Sci. 8 (2015) 983e994. https://doi.org/10.1039/C4EE03157A. [84] D. Xue, P.V. Balachandran, J. Hogden, J. Theiler, D. Xue, T. Lookman, Accel-
[55] Prashun Gorai, Duanfeng Gao, Brenden Ortiz, Sam Miller, Scott A. Barnett, erated search for materials with targeted properties by adaptive design, Nat.
Thomas Mason, Lv Qin, Vladan Stevanovic, S. Eric, T.E. Toberer, Design Lab: a Commun. 7 (2016) 11241.
virtual laboratory for thermoelectric material design, Comp. Mat. Sci. 112 [85] M.A. Clyde, Experimental design: a Bayesian perspective, 2007. http://
(2016) 368e376. www.tedesignlab.org. citeseerx.ist.psu.edu/viewdoc/summary?doi¼10.1.1.28.6849.
[56] W.W. Tipton, C.R. Bealing, K. Mathew, R.G. Hennig, Structures, phase sta- [86] E. Brochu, V.M. Cora, N. de Freitas, ArXiv e-prints, 2010. arXiv:1012.2599.
bilities and electrical potentials of Li-Si battery anode materials, Phys. Rev. B [87] K. Barmak, A. Darbal, K.J. Ganesh, P.J. Ferreira, J.M. Rickman, T. Sun, B. Yao,
87 (2013) 184114. A.P. Warren, K.R. Coffey, Surface and grain boundary scattering in nano-
[57] L. Cheng, R.S. Assary, X. Qu, A. Jain, S.P. Ong, N.N. Rajput, K. Persson, metric Cu thin films: a quantitative analysis including twin boundaries,
L.A. Curtiss, Accelerating electrolyte discovery for energy storage with high- J. Vac. Sci. Technol. A 32 (2014), 061503.
throughput screening, J. Phys. Chem. Lett. 6 (2015) 283e291. [88] R. Howard, Information value theory, IEEE Trans. Syst. Sci. Cybern. 2 (1966)
[58] A.A. Emery, J.E. Saal, S. Kirklin, V.I. Hegde, C. Wolverton, High-throughput 22e26.
DFT calculations of formation energy, stability and oxygen vacancy forma- [89] P. Frazier, W. Powell, S. Dayanik, The knowledge gradient policy for corre-
tion energy of ABO3 perovskites, Chem. Mater. 28 (2016) 5621e5634. lated normal beliefs, Inf. J. Comput. 21 (2009) 599.
[59] W. Sun, S.T. Dacek, S.P. Ong, G. Hautier, A. Jain, W.D. Richards, A.C. Gamst, [90] H. Kushner, A new method of locating the maximum point of an arbitrary
K.A. Persson, G. Ceder, The thermodynamic scale of inorganic crystalline multipeak curve in the presence of noise, J. Basic Eng. 86 (1964) 97e106.
metastability, Sci. Adv. 2 (2016), e1600225. [91] J. Mockus, V. Tiesis, A. Zilinskas, in: L. Dixon, G. Szego (Eds.), Towards Global
[60] F. Legrain, J. Carrete, A. van Roekeghem, G.K.H. Madsen, N. Mingo, Materials Optimization 2, Elsevier Science Ltd., North Holland, Amsterdam, 1978,
screening for the discovery of new half-Heuslers: machine learning versus ab pp. 117e129.
initio methods, J. Phys. Chem. 122 (2018) 625. [92] D.R. Jones, M. Schonlau, W.J. Welch, Efficient global optimization of expen-
[61] P.V. Balachandran, A.E. Emory, J.E. Gubernatis, T. Lookman, C. Wolverton, sive black-box functions, J. Global Optim. 13 (1998) 455e492.
A. Zunger, Predictions of new ABO3 perovskite compounds by combining [93] M. Tokic, Adaptive epsilon-greedy exploration in reinforcement learning
machine learning and density functional theory, Phys. Rev. M 2 (2018) based on value differences, in: R. Dillmann, J. Beyerer, U.D. Hanebeck,
043802. T. Schultz (Eds.), KI 2010: Advances in Artificial Intelligence. KI 2010. Lecture
[62] D. Wu, L. Wu, D. He, L.D. Zhao, W. Li, M. Wu, M. Jin, J. Xu, J. Jiang, L. Huang, Notes in Computer Science, vol 6359, Springer, Berlin, Heidelberg, 2010,
Y. Zhu, M. Kanatzidis, J. He, Direct observation of vast off-stoichiometric 2010.
defects in single crystalline SnSe, NanoEnergy 35 (2017) 321e330. [94] J. Theiler, B.G. Zimmer, Selecting the selector: comparison of update rules for
[63] S. Lu, Q. Zhou, Y. Ouyang, Y. Guo, Q. Li, J. Wang, Accelerated discovery of discrete global optimization, Stat. Anal. Data Min.: ASA Data Sci. J. 10 (2017)
stable lead-free hybrid organic-inorganic perovskites via machine learning, 211.
Nat. Commun. 9 (2018) 3405. [95] B.-J. Yoon, X. Qian, E.R. Dougherty, Quantifying the objective cost of uncer-
[64] R. Armiento, B. Kozinsky, M. Fornari, G. Ceder, Screening for high- tainty in complex dynamical systems, IEEE Trans. Signal Process. 61 (2013)
performance piezoelectrics using high-throughput density functional the- 2256.
ory, Phys. Rev. B 84 (2011) 014103. [96] S. Chen, K. Reyes, et al., Optimal learning in experimental design using the
[65] R. Armiento, B. Kozinsky, G. Hautier, M. Fornari, G. Ceder, High- throughput knowledge gradient policy with application to characterizing nanoemulsion
screening of perovskite alloys for piezoelectric performance and thermo- stability, SIAM J. Uncertain. Quantification (2015), https://doi.org/10.1137/
dynamic stability, Phys. Rev. B 89 (2014) 134103. 140971129.
[66] J.A. Barr, F.-Y. Lin, M. Ashton, R.G. Hennig, S.B. Sinnott, High-throughput [97] P. V. Balachandran, D. Xue, J. Theiler, J. Hogden and T. Lookman, Adaptive
density functional calculations to optimize properties and interfacial strategies for materials design using uncertainties, Sci. Rep. j 6:19660 j DOI:
chemistry of piezoelectric materials, Phys. Rev. Mater. 2 (2018) 025002. 10.1038/srep19660..
[67] D. Xue, P.V. Balachandran, R. Yuan, T. Hu, X. Qian, E.R. Dougherty, [98] M.F. Cover, O. Warschkow, M.M.M. Bilek, D.R.A. McKenzie, A comprehensive
Turab Lookman, Accelerated search for BaTiO3-based piezoelectrics with survey of M2AX phase elastic properties, J. Phys. Condens. Matter 21 (2009)
vertical morphotropic phase boundary using Bayesian learning, Proc. Natl. 305403.
Acad. Sci. Unit. States Am. (2016) 201607412. [99] T. Ueno, T.D. Rhone, Z. Hou, T. Mizoguch, K. Tsuda Combo, An efficient
[68] E. Perim, D. Lee, Y. Liu, C. Toher, P. Gong, Y. Li, W.N. Simmons, O. Levy, Bayesian optimization library for materials science, Mater. Discov. 4 (2016)
J.J. Vlassak, J. Schroers, S. Curtarolo, Spectral descriptors for bulk metallic 18e21.
glasses based on the thermodynamics of competing crystalline phases, Nat. [100] D. Xue, D. Xue, Y. Ruihao, et al., An informatics approach to transformation
Commun. 7 (2016) 12315. temperatures of NiTi-based shape memory alloys, Acta Mater. 125 (2017)
[69] C. Oses. C. Toher and S. Curtarolo, Autonomous Data-Driven Design of 532e541.
Inorganic Materials with AFLOW, 1803.05035v2. [101] Fang Ren, Logan Ward, T. Williams, K.J. Laws, C. Wolverton, J. Hattrick-
[70] A. Jain, J.A. Bollinger, T.M. Truskett, Inverse Methods for Material Design, Simpers, A. Mehta, Accelerated discovery of metallic glasses through itera-
AICHE, 2014. tion of machine learning and high-throughput experiments, Sci. Adv. 4
[71] A. Zunger, Inverse design in search of materials with target functionalities, (2018) eaaq1566.
 J.M. Rickman et al. / Acta Materialia 168 (2019) 473e510 507

[102] R. Aggarwal, M. Demkowicz, Y.M. Marzouk, Information driven experimental Lett. 11 (2011) 828e834, https://doi.org/10.1021/nl1041808.
design in materials science, in: T. Lookman, F.J. Alexander, K. Rajan (Eds.), [128] C.L. Jia, S.B. Mi, M. Faley, U. Poppe, J. Schubert, K. Urban, Oxygen octahedron
Information Science Materials Discovery Design, vol. 225, Springer Interna- reconstruction in the SrTiO(3)/LaAlO(3) heterointerfaces investigated using
tional Publishing, 2016, pp. 13e44, https://doi.org/10.1007/978-3-319- aberration-corrected ultrahigh-resolution transmission electron microscopy,
23871-5. Phys. Rev. B 79 (2009) 081405, https://doi.org/10.1103/PhysRevB.79.081405.
[103] D. Xue, P.V. Balachandran, H. Wu, Material descriptors for morphotropic [129] Q. He, R. Ishikawa, A. Lupini, L. Qiao, E.J. Moon, O. Ovichinnikov, S.J. May,
phase boundary curvature in lead-free piezoelectrics, Appl. Phys. Lett. 111 M.D. Biegalski, A.Y. Borisevich, Towards 3D mapping of BO6 octahedron
(2017) 032907. rotations at perovskite heterointerfaces, unit cell by unit cell, ACS Nano 9
[104] R. Yuan, Z. Liu, et al., Accelerated discovery of large electrostrains in BaTiO3- (2015) 8412e8419, https://doi.org/10.1021/acsnano.5b03232.
based piezoelectrics using active learning, Adv. Mater. 30 (7) (2018) [130] Y.M. Kim, A. Kumar, A. Hatt, A. Morozovska, A. Tselev, M.D. Biegalski,
1702884. I. Ivanov, E. Eliseev, S.J. Pennycook, J.M. Rondinelli, S.V. Kalinin,
[105] P.V. Balachandran, B. Kowalski, A. Sehirlioglu, T. Lookman, Experimental A.Y. Borisevich, Interplay of octahedral tilts and polar order in BiFeO3 films,
search for high-temperature ferroelectric perovskites guided by two-step Adv. Mater. 25 (2013) 2497e2504, https://doi.org/10.1002/
machine learning, Nat. Commun. 9 (2018) 1668. adma.201204584.
[106] Y. Wang, Y. Zhanga, H. Zhao, X. Li, Y. Huang, L.S. Schadler, W. Chen, [131] A.Y. Borisevich, H.Y. Chang, M. Huijben, M.P. Oxley, S. Okamoto, Suppression
L. Catherine Brinson, Identifying interphase properties in polymer nano- of octahedral tilts and associated changes in electronic properties at epitaxial
composites using adaptive Optimization, Compos. Sci. Technol. 162 (2018) oxide heterostructure interfaces, Phys. Rev. Lett. 105 (2010) 087204, https://
146e155. doi.org/10.1103/PhysRevLett.105.087204.
[107] B. Rouet-Leduc, K. Barros, T. Lookman, C.J. Humphreys, Optimization of GaN [132] J. He, A. Borisevich, S.V. Kalinin, S.J. Pennycook, Pantelides, Control of octa-
LEDs and the reduction of efficiency droop using active machine learning, hedral tilts and magnetic properties of perovskite oxide heterostructures by
Sci. Rep. 6 (2016) 24862. substrate symmetry, Phys. Rev. Let. 105 (2010) 227203, https://doi.org/
[108] A.J. Keane, Statistical improvement criteria for use in multiobjective design 10.1103/PhysRevLett.105.227203, 2010.
optimization, AIAA J. 44 (2006) 879. [133] Y.M. Kim, A. Morozovska, E. Eliseev, M.P. Oxley, R. Mishra, S.M. Selbach,
[109] J. Svenson, T. Santner, Multiobjective optimization of expensive-to- evaluate T. Grande, S.T. Pantelides, S.V. Kalinin, A. Borisevich, Direct observation of
deterministic computer simulator models, Comput. Stat. Data Anal. 94 ferroelectric field effect and vacancy-controlled screening at the BiFeO3/
(2016) 250. LaxSr1-xMnO3 interface, Nat. Mater. 13 (2014) 1019e1025, https://doi.org/
[110] A.M. Gopakumar, P.V. Balachandran, D. Xue, J.E. Gubernatis, T. Lookman, 10.1038/nmat4058.
Multiobjective optimization for materials discovery via adaptive design, Sci. [134] Y.M. Kim, J. He, M. Biegalski, H. Ambaye, V. Lauter, H.M. Christen,
Rep. 8 (2018) 3738. S.T. Pantelides, S.J. Pennycook, S.V. Kalinin, A. Borisevich, Probing oxygen
[111] M. Emmerich, N. Beume, B. Naujoks, An EMO algorithm using the hyper- vacancy concentration and homogeneity in solid-oxide fuel-cell cathode
volume measure as selection criterion, in: EMO, Vol. 3410 of Lecture Notes in materials on the subunit-cell level, Nat. Mater. 11 (2012) 888e894, https://
Computer Science, Springer, 2005, pp. 62e76. doi.org/10.1038/nmat3393.
[112] L. Lu, C.M. Anderson-Cook, Adapting the hypervolume quality indicator to [135] A. Belianinov, Q. He, M. Kravchenko, S. Jesse, A. Borisevich, S.V. Kalinin,
quantify trade-offs and search efficiency for multiple criteria decision mak- Identification of phases, symmetries and defects through local crystallog-
ing using Pareto fronts, Qual. Reliab. Eng. Int. 29 (2012) 1117. raphy, Nat. Commun. 6 (2015) 7801, https://doi.org/10.1038/ncomms8801.
[113] Y. Cao, B.J. Smucker, T.J. Robinson, On using the hypervolume indicator to [136] Z. Gai, W. Lin, J.D. Burton, K. Fuchigami, P.C. Snijders, Chemically induced
compare Pareto fronts: applications to multicriteria optimal experimental Jahn-Teller ordering on manganite surfaces, Nat. Commun. 5 (2014) 4528,
design, J. Stat. Plan. Infer. 160 (2015) 60. https://doi.org/10.1038/ncomms5528.
[114] A. Talapatra, S. Boluki, T. Duong, X. Qian, E. Dougherty, R. Arroyave, Auton- [137] O.L. Krivanek, T.C. Lovejoy, N. Dellby, T. Aoki, R.W. Carpenter, P. Rez,
omous efficient experiment design for materials discovery with Bayesian E. Soignard, J. Zhu, P.E. Batson, M.J. Lagos, R.F. Egerton, P.A. Crozier, Vibra-
model averaging, Phys. Rev. Mater. 2 (2018) 113803. tional spectroscopy in the electron microscope, Nature 514 (2014) 209,
[115] A. Solomou, G. Zhao, S. Boluki, J.K. Joy, D.C. Lagoudas, Multi-objective https://doi.org/10.1038/nature13870.
Bayesian materials discovery: application of the discovery of precipitation [138] M.F. Chisholm, W.D. Luo, M.P. Oxley, S.T. Pantelides, H.N. Lee, Atomic-scale
strengthened NiTi shape memory alloys through micromechanical modeling, compensation phenomena at polar interfaces, Phys. Rev. Lett. 105 (2010)
Mater. Des. 160 (2018) 810e827. 197602, https://doi.org/10.1103/PhysRevLett.105.197602.
[116] G. Zhao, R. Arroyave and X. Qian, 1812.07692 (2018). [139] A.Y. Borisevich, A. Lupini, J. He, E. Eliseev, A. Morozovska, G. Svechnikov,
[117] G. Pilania, J.E. Gubernatis, T. Lookman, Multi-fidelity machine learning P. Yu, Y.-H. Chu, R. Ramesh, S.T. Pantelides, S.V. Kalinin, S.J. Pennycook,
models for accurate bandgap predictions of solids, Comp. Mat. Sci. 129 Interface dipole between two metallic oxides caused by localized oxygen
(2017) 156. vacancies, Phys. Rev. B 86 (2012) 140102, https://doi.org/10.1103/
[118] D.H. Wolpert, W.G. Macready, No free lunch theorems for optimization, IEEE PhysRevB.86.140102.
Trans. Evol. Comput. 1 (1997) 67. [140] P. Perna, D. Maccariello, M. Radovic, U. Uccio, I. Pallecchi, M. Codda, D. Marre,
[119] S.J. Pennycook, P.D. Nellist, Scanning Transmission Electron Microscopy, C. Cantoni, J. Gasquez, M. Varela, S. Pennycook, F. Granozio, Conducting in-
Springer, New York, 2011. terfaces between band insulating oxides: the LaGaO(3)/SrTiO(3) hetero-
[120] C. Gerber, H.P. Lang, How the doors to the nanoworld were opened, Nat. structure, Appl. Phys. Lett. 97 (2010) 152111, https://doi.org/10.1063/
Nanotechnol. 1 (2006) 3e5, https://doi.org/10.1038/nnano.2006.70. 1.3496440.
[121] A.B. Yankovic, B. Berkels, W. Dahmen, P. Binev, S.I. Sanchev, S.A. Bradley, [141] R.K. Vasudevan, M. Ziatdinov, S. Jesse, S.V. Kalinin, Phases and interfaces
A. Li, I. Szlufarska, P.M. Voyles, Picometre-precision analysis of scanning from real space atomically resolved data: physics-based deep data image
transmission electron microscopy images of platinum nanocatalysts, Nat. analysis, Nano Lett. 16 (2016) 5574e5581, https://doi.org/10.1021/
Commun. 5 (2014) 4155, https://doi.org/10.1038/ncomms5155. acs.nanolett.6b02130.
[122] C.L. Jia, V. Nagarajan, J.-Q. He, L. Houben, T. Zhao, R. Ramesh, K. Urban, [142] Q. He, J. Woo, A. Belianinov, V.V. Guliants, A.Y. Borisevich, Better catalysts
R. Waser, Unit-cell scale mapping of ferroelectricity and tetragonality in through microscopy: mesoscale M1/M2 intergrowth in molybdenum-
epitaxial ultrathin ferroelectric films, Nat. Mater. 6 (2007) 64e69, https:// vanadium based complex oxide catalysts for propane ammoxidation, ACS
doi.org/10.1038/nmat1808. Nano 9 (2015) 3470e3478, https://doi.org/10.1021/acsnano.5b00271.
[123] C.L. Jia, S.B. Mi, K. Urban, I. Vrejoiu, M. Alexe, D. Hesse, Atomic-scale study of [143] A. Kumar, O. Ovichinnikov, S. Guo, F. Griggio, S. Jesse, S. Trolier-McKinstry,
electric dipoles near charged and uncharged domain walls in ferroelectric S.V. Kalinin, Spatially resolved mapping of disorder type and distribution in
films, Nat. Mater. 7 (2008) 57e61, https://doi.org/10.1038/nmat2080. random systems using artificial neural network recognition, Phys. Rev. B 84
[124] A. Borisevich, O.S. Ovchinnikov, H.J. Chang, M.P. Oxley, P. Yu, J. Seidel, (2011) 024203, https://doi.org/10.1103/PhysRevB.84.024203.
E. Eliseev, A.N. Morozovska, R. Ramesh, S.J. Pennycook, S.V. Kalinin, Mapping [144] S.V. Kalinin, A.N. Morozovska, L.-Q. Chen, B.J. Rodriguez, Local polarization
octahedral tilts and polarization across a domain wall in BiFeO(3) from Z- dynamics in ferroelectric materials, Rep. Prog. Phys. 73 (2010) 056502,
Contrast scanning transmission electron microscopy image atomic column https://doi.org/10.1088/0034-4885/73/5/056502.
shape analysis, ACS Nano 4 (2010) 6071e6079, https://doi.org/10.1021/ [145] B.J. Rodriguez, S. Jesse, J. Kim, S. Ducharme, S.V. Kalinin, Local probing of
nn1011539. relaxation time distributions in ferroelectric polymer nanomesas: time-
[125] Q. Li, C.T. Nelson, S.-L. Hsu, A.R. Damodaran, L.-L. Li, A.K. Yadav, M. McCarter, resolved piezoresponse force spectroscopy and spectroscopic imaging,
L.W. Martin, R. Ramesh, S.V. Kalinin, Quantification of flexoelectricity in Appl. Phys. Lett. 92 (2008) 232903, https://doi.org/10.1063/1.2942390.
PbTiO3/SrTiO3 superlattice polar vortices using machine learning and phase- [146] S. Jesse, P. Maksymovych, S.V. Kalinin, Rapid multidimensional data acqui-
field modeling, Nat. Commun. 8 (2017) 1468, https://doi.org/10.1038/ sition in scanning probe microscopy applied to local polarization dynamics
s41467-017-01733-8. and voltage dependent contact mechanics, Appl. Phys. Lett. 93 (2008)
[126] C.T. Nelson, P. Gao, J.R. Jokisaari, C. Heikes, C. Adamo, A. Melville, S.-H. Baek, 112903, https://doi.org/10.1063/1.2980031.
C.M. Folkman, B. Winchester, Y. Gu, Y. Liu, K. Zhang, E. Wang, J. Li, L.-Q. Chen, [147] R.K. Vasudevan, S. Jesse, Y. Kim, A. Kumar, S.V. Kalinin, Spectroscopic im-
C.-B. Eom, D.G. Schlom, X. Pan, Domain dynamics during ferroelectric aging in piezoresponse force microscopy: new opportunities for studying
switching, Science 334 (2011) 968e971, https://doi.org/10.1126/ polarization dynamics in ferroelectrics and multiferroics, MRS Commun. 2
science.1206980. (2012) 61e73, https://doi.org/10.1557/mrc.2012.15.
[127] C.T. Nelson, B. Winchester, Y. Zhang, S.-J. Kim, A. Melville, C. Adamo, [148] S. Jess, S.V. Kalinin, Principal component and spatial correlation analysis of
C.M. Folkman, S.-H. Baek, C.-B. Eom, D.G. Schlom, L.-Q. Chen, X. Pan, Spon- spectroscopic-imaging data in scanning probe microscopy, Nanotechnology
taneous vortex nanodomain arrays at ferroelectric heterointerfaces, Nano 20 (2009) 085714, https://doi.org/10.1088/0957-4484/20/8/085714.
508 J.M. Rickman et al. / Acta Materialia 168 (2019) 473e510

[149] S.V. Kalinin, E. Strelcov, A. Belianinov, S. Somnath, R.K. Vasudevan, platinum nanostructures using in situ liquid cell helium ion microscopy,
E.J. Lingerfelt, R.K. Archibald, C. Chen, R. Proksch, N. Laanait, Jesse Stephen, Nanoscale 9 (2017) 12949e12956, https://doi.org/10.1039/c7nr04417h.
Big, deep, and smart data in scanning probe microscopy, ACS Nano 10 (2016) [170] William A.M. Moeglein, Richard L. Griswold, Bourgeois Mehdi, Nigel
9068e9086, https://doi.org/10.1021/acsnano.6b04212. D. Browning, Jeremy Teuton, Applying shot boundary detection for auto-
[150] R.K. Vasudevan, A. Belianinov, A.G. Gianfrancesco, A.P. Baddorf, A. Tselev, mated crystal growth analysis during in situ transmission electron micro-
S.V. Kalinin, S. Jesse, Big data in reciprocal space: sliding fast Fourier trans- scope experiments, Adv. Struct. Chem. Imag. 3 (2017) 2, https://doi.org/
forms for determining periodicity, Appl. Phys. Lett. 106 (2015), https:// 10.1186/s40679-016-0034-x.
doi.org/10.1063/1.4914016, 091601. [171] A.V. Ievlev, S. Jesse, T. Cochell, R. Unocic, V.A. Protopopescu, S.V. Kalinin,
[151] E. Strelcov, A. Belianin, Y.-H. Hsieh, Y. Chu, S.V. Kalinin, Constraining data Quantitative description of crystal nucleation and growth from in situ
mining with physical models: voltage- and oxygen pressure-dependent scanning transmission electron microscopy, ACS Nano 9 (2015)
transport in multiferroic nanostructures, Nano Lett. 15 (2015) 6650e6657, 11784e11791, https://doi.org/10.1021/acsnano.5b03720.
https://doi.org/10.1021/acs.nanolett.5b02472. [172] I.I. Naumov, L. Bellaiche, H.X. Fu, Unusual phase transitions in ferroelectric
[152] M. Ziatdinov, A. Maksov, L. Li, S. Sefat, P. Maksymovych, S.V. Kalinin, Deep nanodisks and nanorods, Nature 432 (2004) 737e740, https://doi.org/
data mining in a real space: separation of intertwined electronic responses in 10.1038/nature03107.
a lightly doped BaFe2As2, Nanotechnology 27 (2016) 475706. [173] Bo-Kuai Lai, I. Ponomareva, I.I. Naumov, I. Kornev, Huaxiang Fu, L. Bellaiche,
[153] Maxim Ziatdinov, Shintaro Fujii, Manabu Kiguchi, Toshiaki Enoki, G.J. Salamo, Electric-field-induced domain evolution in ferroelectric ultrathin
Jesse Stephen, Sergei Kalinin, Data mining graphene: correlative analysis of films, Phys. Rev. Lett. 96 (2006) 137602, https://doi.org/10.1103/
structure and electronic degrees of freedom in graphenic monolayers with PhysRevLett.96.137602.
defects, Nanotechnology 27 (2016) 495703. [174] I. Ponomareva, I.I. Naumov, I. Kornev, Huaxiang Fu, L. Bellaiche, Atomistic
[154] M. Ziatdinov, A. Maksov, S.V. Kalinin, Learning surface molecular structures treatment of depolarizing energy and field in ferroelectric nanostructures,
via machine vision, npj Comput. Mater. 3 (2017) 31, https://doi.org/10.1038/ Phys. Rev. B 72 (2005) 140102, https://doi.org/10.1103/PhysRevB.72.140102.
s41524-017-0038-7. [175] C.L. Jia, K.W. Urban, M. Alexe, D. Hesse, I. Vrejoiu, Direct observation of
[155] M.P. Nikiforov, V.V. Reukov, G.L. Thompson, A.A. Vertegel, S. Guo, S. Jesse, continuous electric dipole rotation in flux-closure domains in ferroelectric
S.V. Kalinin, Functional recognition imaging using artificial neural networks: Pb(Zr,Ti)O(3), Science 331 (2011) 1420e1423, https://doi.org/10.1126/
applications to rapid cellular identification via broadband electromechanical science.1200605.
response, Nanotechnology 20 (2009) 405708, https://doi.org/10.1088/0957- [176] Y. Ivry, D.P. Chu, J.F. Scott, C. Durkan, Flux closure vortex like domain
4484/20/40/405708. structures in ferroelectric thin films, Phys. Rev. Lett. 104 (2010) 20760,
[156] O.S. Ovchinnikov, S. Jesse, P. Bintacchit, S. Trolier-McKinstry, S.V. Kalinin, https://doi.org/10.1103/PhysRevLett.104.207602.
Disorder identification in hysteresis data: recognition analysis of the [177] N. Balke, B. Winchester, W. Ren, Y.H. Chu, A. Morozovska, E. Eliseev,
random-bond-random-field Ising model, Phys. Rev. Lett. 103 (2009) 157203, M. Huijben, R. Vasudevan, P. Maksymovych, J. Britson, S. Jesse, I. Kornev,
https://doi.org/10.1103/PhysRevLett.103.157203. R. Ramesh, L. Bellaiche, L.-Q. Chen, S.V. Kalinin, Enhanced electric conduc-
[157] M. Ziatdinov, O. Dyck, A. Maksov, X. Li, X. Sang, K. Xiao, R.R. Unocic, tivity at ferroelectric vortex cores in BiFeO(3), Nat. Phys. 8 (2012) 81e88.
R.K. Vasudevan, S. Jesse, S.V. Kalinin, Deep learning of atomically resolved [178] A.K. Yadav, C.T. Nelson, S.L. Hsu, Z. Hong, J.D. Clarkson, C.M. Schlepüetz,
scanning transmission electron microscopy images: chemical identification A.R. Damodaran, P. Shafer, E. Arenholz, L.R. Dedon, D. Chen, A. Vishwanath,
and tracking local transformations, ACS Nano 11 (2017) 12742. A.M. Minor, L.Q. Chen, J.F. Scott, L.W. Martin, R. Ramesh, Observation of polar
[158] Maxim Ziatdinov, Ondrej Dyck, Artem Maksov, Bethany M. Hudak, Andrew vortices in oxide superlattices, Nature 530 (2016) 198, https://doi.org/
R. Lupini, Jiaming Song, Paul C. Snijders, Rama K. Vasudevan, Jesse Stephen, 10.1038/nature16463.
Sergei V. Kalinin, Deep analytics of atomically-resolved images: manifest and [179] E. Eliseev, A.N. Morozovska, G.S. Svechnikov, P. Maksymovych, S.V. Kalinin,
latent features. arXiv:1801.05133, 2018. Domain wall conduction in multiaxial ferroelectrics, Phys. Rev. B 85 (2012)
[159] Jacob Madsen, Pei Liu, Jens Kling, Jakob Birkedal Wagner, Thomas 045312, https://doi.org/10.1103/PhysRevB.85.045312.
Willum Hansen, Ole Winther, Jakob Schiøtz, A deep learning approach to [180] Eugene A. Eliseev, Anna N. Morozovska, Yijia Gu, Albina Y. Borisevich, Long-
identify local structures in atomic-resolution transmission electron micro- Qing Chen, Venkatraman Gopalan, Sergei V. Kalinin, Conductivity of twin-
scopy images, Adv. Theor. Simulat. 0 (2018) 1800037, https://doi.org/ domain-wall/surface junctions in ferroelastics: interplay of deformation
10.1002/adts.201800037. potential, octahedral rotations, improper ferroelectricity, and flexoelectric
[160] Artem Maksov, Ondrej Dyck, Kai Wang, Kai Xiao, David B. Geohegan, Bobby coupling, Phys. Rev. B 86 (2012) 085416, https://doi.org/10.1103/
G. Sumpter, Rama K. Vasudevan, Jesse Stephen, Sergei V. Kalinin, PhysRevB.86.085416.
Maxim Ziatdinov, Deep Learning Analysis of Defect and Phase Evolution [181] A.N. Morozovska, R.K. Vasudevan, P. Maksymovych, S.V. Kalinin, E.A. Eliseev,
during Electron Beam Induced Transformations in WS2, 2018. arXiv:1803. Anisotropic conductivity of uncharged domain walls in BiFeO3, Phys. Rev. B
05381. 86 (2012), https://doi.org/10.1103/PhysRevB.86.085315.
[161] Lucas R. Parent, David B. Robinson, Taylor J. Woehl, William D. Ristenpart, [182] A.N. Morozovska, E.A. Eliseev, S.V. Kalinin, L.Q. Chen, V. Gopalan, Surface
James E. Evans, Nigel D. Browning, Ilke Arslan, Direct in situ observation of polar states and pyroelectricity in ferroelastics induced by flexo-roto field,
nanoparticle synthesis in a liquid crystal surfactant template, ACS Nano 6 Appl. Phys. Lett. 100 (2012) 142902, https://doi.org/10.1063/1.3701152.
(2012) 3589e3596, https://doi.org/10.1021/nn300671g. [183] A.N. Morozovska, E. Eliseev, M.D. Glinchuk, L.Q. Chen, V. Gopalan, Interfacial
[162] T.J. Woehl, J.E. Evans, I. Arslan, W.D. Ristenpart, N.D. Browning, Direct in situ polarization and pyroelectricity in antiferrodistortive structures induced by a
determination of the mechanisms controlling nanoparticle nucleation and flexoelectric effect and rotostriction, Phys. Rev. B 85 (2012) 094107, https://
growth, ACS Nano 6 (2012) 8599e8610, https://doi.org/10.1021/nn303371y. doi.org/10.1103/PhysRevB.85.094107.
[163] T.J. Woehl TJ, C. Park, J.E. Evans, I. Arslan, W.D. Ristenpart, N.D. Browning, [184] A.K. Tagantsev, Piezoelectricity and flexolectricity in crystalline dielectrics,
Direct observation of aggregative nanoparticle growth: kinetic modeling of Phys. Rev. B 34 (1986) 5883e5889, https://doi.org/10.1103/
the size distribution and growth rate, Nano Lett 14 (2014) 373e378, https:// PhysRevB.34.5883.
doi.org/10.1021/nl4043328. [185] Eugene A. Eliseev, Sergei V. Kalinin, Yijia Gu, Maya D. Glinchuk,
[164] J. Yang, C.M. Andrei, G. Botton, L. Soleymani, In liquid observation and Victoria Khist, Albina Borisevich, Venkatraman Gopalan, Long-Qing chen,
quantification of nucleation and growth of gold nanostructures using in situ Anna N. Morozovska, Universal emergence of spatially modulated structures
transmission electron microscopy, J. Phys. Chem. C 121 (2017) 7435e7441, induced by flexo antiferrodistortive coupling in multiferroics, Phys. Rev. B 88
https://doi.org/10.1021/acs.jpcc.6b10400. (2013) 224105, https://doi.org/10.1103/PhysRevB.88.224105.
[165] Boao Song, Kun He, Yifei Yuan, Soroosh Sharifi-Asl, Meng Cheng, Jun Lu, [186] A.Y. Borisevich, E.A. Eliseev, A.N. Morozovska, C.-J. Cheng, J.-Y. Lin, Y.H. Chu,
Wissam A. Saidi, Reza Shahbazian-Yassar, In situ study of nucleation and D. Kan, I. Takeuchi, V. Nagarajan, S.V. Kalinin, Atomic-scale evolution of
growth dynamics of Au nanoparticles on MoS2 nanoflakes, Nanoscale 10 modulated phases at the ferroelectric-antiferroelectric morphotropic phase
(2018) 15809e15818, https://doi.org/10.1039/C8NR03519A. boundary controlled by flexoelectric interaction, Nat. Commun. 3 (2012) 77,
[166] Michael P. Brady, Anton V. Ievlev, Mostafa Fayek, Donovan N. Leonard, https://doi.org/10.1038/ncomms1778.
Matthew G. Frith, M. Harry, Meyer III, J. Anibal, Ramirez-Cuesta, Luke [187] A.Y. Borisevich, A.N. Morozovska, Young-Min Kim, D. Leonard, M.P. Oxley,
L. Daemen, Yongqiang Cheng, W. Gao, J. Poplawsky, O. Ovchinnikova, M.D. Biegalski, E.A. Eliseev, S.V. Kalinin, Exploring mesoscopic physics of
J. Thomson, L.M. Anovitz, G. Rother, D. Shin, G.-L. Song, B. Davis, Rapid vacancy-ordered systems through atomic scale observations of topological
diffusion and nanosegregation of hydrogen in magnesium alloys from defects, Phys. Rev. Lett. 109 (2012) 065702, https://doi.org/10.1103/
exposure to water, ACS Appl. Matt. Interf. 9 (2017) 38125e38134, https:// PhysRevLett.109.065702.
doi.org/10.1021/acsami.7b10750. [188] L. Vlcek, A. Maksov, M.H. Pan, R.K. Vasudevan, S.V. Kalinin, Knowledge
[167] Jeung Hun Park, Nicholas M. Schneider, Joseph M. Grogan, Mark C. Reuter, extraction from atomically resolved images, ACS Nano 11 (2017)
Haim H. Bau, Suneel Kodambaka, Frances M. Ross, Control of electron beam- 10313e10320, https://doi.org/10.1021/acsnano.7b05036.
induced Au nanocrystal growth kinetics through solution chemistry, Nano [189] L. Vlcek, W.W. Sun, P.R.C. Kent, Combining configurational energies and
Lett. 15 (2015) 5314e5320, https://doi.org/10.1021/acs.nanolett.5b01677. forces for molecular force field optimization, J. Chem. Phys. 147 (2017)
[168] N.M. Schneider, J.H. Park, M.M. Norton, F.M. Ross, H.H. Bau, Automated 161713, https://doi.org/10.1063/1.4986079.
analysis of evolving interfaces during in situ electron microscopy, Adv. [190] L. Vlcek, R.K. Vasudevan, S. Jesse, S.V. Kalinin, Consistent integration of
Struct. Chem. Imag. 2 (2016) 2, https://doi.org/10.1186/s40679-016-0016-z. experimental and ab initio data into effective physical models, J. Chem.
[169] Anton V. Ievlev, Jacek Jakowski, Matthew J. Burch, Vighter Iberi, Holland Theor. Comput. 13 (2017) 5179e5194, https://doi.org/10.1021/
E. Hysmith, David C. Joy, Bobby G. Sumpter, Alexei Belianinov, Raymond acs.jctc.7b00114.
R. Unocic, Olga S. Ovchinnikova, Building with ions: towards direct write of [191] D. Collin, F. Ritort, C. Jarzynski, S.B. Smith, I. Tinoco Jr., C. Bustamante,
 J.M. Rickman et al. / Acta Materialia 168 (2019) 473e510 509

Verification of the Crooks fluctuation theorem and recovery of RNA folding correlated nuclei, Phys. Rev. E 95 (2017), 022121.
free energies, Nature 437 (2005) 231. [224] N.G. van Kampen, Stochastic Processes in Physics and Chemistry, third ed.,
[192] N.C. Harris, Y. Song, C.-H. Kiang, Experimental free energy surface recon- Elsevier, Amsterdam, 2007.
struction from single-molecule force spectroscopy using Jarzynski's equality, [225] J.A. Lee, M. Verleysen, Nonlinear Dimensionality Reduction, Springer, New
Phys. Rev. Lett. 99 (2007) 068101. York, 2007.
[193] Jesse Stephen, Albina Y. Borisevich, Jason D. Fowlkes, Andrew R. Lupini, [226] J.E. Jackson, A User's Guide to Principal Components, John Wiley and Sons,
Philip D. Rack, Raymond R. Unocic, Bobby G. Sumpter, Sergei V. Kalinin, Hoboken, NJ, 2003.
Alex Belianinov, Olga S. Ovchinnikova, Directing matter: toward atomic- [227] T.R. Knapp, Canonical correlation analysis: a general parametric significance-
scale 3D nanofabrication, ACS Nano 10 (2016) 5600e5618, https://doi.org/ testing system, Psych. Bull. 85 (1978) 410e416.
10.1021/acsnano.6b02489. [228] C.J.C. Burges, Dimension reduction: a guided tour, Found. Trends Mach.
[194] D. Ross, E.A. Strychalski, C. Jarzynski, S.M. Stavis, Equilibrium free energies Learn. 2 (2009) 275e365.
from non-equilibrium trajectories with relaxation fluctuation spectroscopy, [229] C. Thornton, Separability is a learner's best friend, in: J.A. Bullinaria, et al.
Nat. Phys. (2018), https://doi.org/10.1038/s41567-018-0153-5. (Eds.), Proceedings of the 4th Neural Computation and Psychology Work-
[195] J. Carrasquilla, R.G. Melko, Machine learning phases of matter, Nat. Phys. 13 shop: Connectionist Representations, Springer-Verlag, 1997.
(2017) 431, https://doi.org/10.1038/nphys4035. https://www.nature.com/ [230] G. Brock, V. Pihur, S. Datta, S. Datta, G. Brock, V. Pihur, S. Datta, cIValid, an R
articles/nphys4035#supplementary-information. package for cluster validation, J. Stat. Softw. 25 (2008) 4.
[196] A. Tanaka, A. Tomiya, Detection of phase transition via convolutional neural [231] A. Inselberg, Parallel Coordinates: VISUAL Multidimensional Geometry and
networks, J. Phys. Soc. Jpn. 86 (2017) 063001, https://doi.org/10.7566/ its Applications, Springer, 2009.
JPSJ.86.063001. [232] D.L. McDowell, R. LeSar, The need for microstructure informatics in process-
[197] K. Ch’ng, J. Carrasquilla, R. Melko, E. Khatami, Machine learning phases of structure-property relations, MRS Bull. 41 (2014) 587e593.
strongly correlated fermions, Phys. Rev. X 7 (2017) 031038, https://doi.org/ [233] S.R. Kalidindi, Hierarchical Materials Informatics: Novel Analytics for Materials
10.1103/PhysRevX.7.031038. Data, Elsevier, Oxford, 2015.
[198] S.J. Wetzel, M. Scherzer, Machine learning of explicit order parameters: from [234] A.C. Lewis, A.B. Geltmacher, Image-based modeling of the response of
the Ising model to SU(2) lattice gauge theory, Phys. Rev. B 96 (2017) 184410, experimental 3D microstructures to mechanical loading, Scr. Mater. 55
https://doi.org/10.1103/PhysRevB.96.184410. (2006) 81.
[199] Y. Zhang, E.A. Kim, Quantum loop topography for machine learning, Phys. [235] A.C. Lewis, C. Suh, M. Stukowski, A.B. Geltmacher, G. Spanos, K. Rajan,
Rev. Lett. 118 (2017) 216401, https://doi.org/10.1103/ Quantitative analysis and feature recognition in 3-D microstructural data
PhysRevLett.118.216401. sets, JOM 58 (2006) 52.
[200] E. van Nieuwenburg, E. Bairey, G. Refael, Learning phase transitions from [236] R. Liu, Z.J. Zhang, L.L. Li, X.H. An, Z.F. Zhang, Microscopic mechanism
dynamics. arXiv:1712.00450, 2017. contributing to the synchronous improvement of strength and plasticity
[201] Q. Wei, R. Melko, J.Z.Y. Chen, Identifying polymer states by machine learning, (SISP) for TWIP copper alloys, Sci. Rep.-UK 5 (2015) 9550.
Phys. Rev. 95 (2017) 032504, https://doi.org/10.1103/PhysRevE.95.032504. [237] S. Keshavarz, S. Ghosh, Hierarchical crystal plasticity FE model for Ni-based
[202] E. van Nieuwenburg, Y.,-H. Liu, S. Huber, Learning phase transitions by superalloys: sub-grain microstructures to polycrystalline aggregates, Int. J.
confusion, Nat. Phys. 13 (2017) 435, https://doi.org/10.1038/nphys4037. Solid Struct. 55 (2015) 17e31.
[203] Y.-H. Liu, E. van Nieuwenburg, Discriminative cooperative networks for [238] Z. Zhang, B.Q. Han, K.H. Chung, E.J. Lavernia, On the behavior of micro-
detecting phase transitions, Phys. Rev. Lett. 120 (2018) 176401, https:// structures with multiple length scales, Metall. Mater. Trans. 37A (2006)
doi.org/10.1103/PhysRevLett.120.176401. 2265e2273.
[204] P. Broecker, F.F. Assaad, S. Trebst, Quantum Phase Recognition via Unsu- [239] S. Torquato, Statistical description of microstructures, Ann. Rev. Mater. Res.
pervised Machine Learning, 2017. arXiv:1707.00663. 32 (2002) 77e111.
[205] D.P. Kingma, M. Welling, Auto-encoding Variational Bayes, 2013. arXiv:1312. [240] D.T. Fullwood, S.R. Niezgoda, S.R. Kalidindi, Microstructure reconstructions
6114. from two-point statistics using phase-recovery algorithms, Acta Mater. 56
[206] C. Doersch, Tutorial on Variational Autoencoders, 2016. arXiv:1606.05908. (2008) 942e948.
[207] S.J. Wetzel, Unsupervised learning of phase transitions: from principal [241] D. Chandler, Introduction to Modern Statistical Mechanics, Oxford University
component analysis to variational autoencoders, Phys. Rev. 96 (2017) Press, Oxford, 1987.
022140, https://doi.org/10.1103/PhysRevE.96.022140. [242] N. Sheehan, S. Torquato, Generating microstructures with specified corre-
[208] T.N. Kipf, M. Welling, Semi-supervised Classification with Graph Convolu- lations, J. Appl. Phys. 89 (2001) 53e60.
tional Networks, 2016. arXiv:1609.02907. [243] B.L. Hansen, B.L. Adams, M.E. Lyon, A.J. Henrie, On the reconstruction of
[209] T. Xie, J.C. Grossman, Crystal graph convolutional neural networks for an polycrystalline microstructures from two-point correlation statistics,
accurate and interpretable prediction of material properties, Phys. Rev. Lett. J. Comput.-Aided Mater. 10 (2005) 163e173.
120 (2018) 145301, https://doi.org/10.1103/PhysRevLett.120.145301. [244] G.J. Schmitz, B. Bottger, M. Apel, J. Eiken, G. Laschet, R. Altenfeld, R. Berger,
[210] L. Li, Y. Yang, D. Zhang, Z.-G. Ye, S. Jesse, S.V. Kalinin, R.K. Vasudevan, Ma- G. Boussinot, A. Viardin, “Towards a metadata schema for the description of
chine learning e enabled identification of material phase transitions based materials e the description of microstructures, Sci. Technol. Adv. Mat. 17
on experimental data: exploring collective dynamics in ferroelectric relax- (2016) 410e430.
ors, Sci. Adv. 4 (2018) eaap. 8672. [245] S.M. Azimi, D. Britz, M. Engstler, M. Fritz, F. Mucklich, Advanced steel
[211] E. Wadhwa, E. Dhaliwal, A Textbook of Engineering Material and Metallurgy, microstructural classification by deep learning methods, Sci. Rep. 8 (2018)
University Science Press, Bangalore, 2008. 2128.
[212] H. Cheng, S.J. Dillon, H.S. Caram, J.M. Rickman, H.M. Chan, M.P. Harmer, The [246] J. Gola, D. Britz, T. Staudt, M. Winter, A.S. Schneider, M. Ludovici, F. Mucklich,
effect of yttrium on oxygen grain boundary transport in polycrystalline Advanced microstructure classification by data mining methods, Comp. Mat.
alumina measured using Ni marker particles”, J. Am. Ceram. Soc. 91 (2008) Sci. 148 (2018) 324e335.
2002e2008. [247] A. Velichko, C. Holzapfel, F. Mucklich, 3D characterization of graphite mor-
[213] J. Cho, C.M. Wang, H.M. Chan, J.M. Rickman, M.P. Harmer, Improved tensile phologies in cast iron, Adv. Eng. Mater. 9 (2007) 39e45.
creep properties of yttrium- and lanthanum-doped alumina: a solid solution [248] J.P. MacSlyne, J.P. Simmons, M. deGraef, On the use of 2-D moment in-
effect, J. Mater. Res. 16 (2001) 425e429. variants for the automated classification of particle shapes, Acta Mater. 56
[214] J. Cho, J.M. Rickman, H.M. Chan, M.P. Harmer, Modeling of grain-boundary (2008) 427e437.
segregation behavior in aluminum oxide, J. Cho, J. M. Rickman, H. M. Chan [249] M.K. Hu, Visual pattern recognition by moment invariants, IEEE Trans.
and M. P. Harmer, J. Am. Ceram. Soc. 83 (2000) 344. Inform. Th. 8 (1962) 179e187.
[215] B.L. DeCost, T. Francis, E.A. Holm, Exploring the microstructure manifold: [250] K. Sekimoto, Evolution of the domain structure during the nucleation-and-
image texture representations applied to ultrahigh carbon steel micro- growth-process with a non-conserved order parameter, Physica 135A
structures, Acta Mater. 133 (2017) 30e40. (1986) 328e346.
[216] H. Tang, C. Basaran, Influence of microstructure coarsening on thermo- [251] A.E. Kolmogorov, Bull. Acad. Nauk. USSR, statistical theory of crystallization
mechanical fatigue behavior of Pb/Sn eutectic solder joints, Int. J. Damage of metals, Mat. Ser. 1 (1937) 355.
Mech. 10 (2001) 235e255. [252] W.A. Johnson, R.F. Mehl, Reaction kinetics in processes of nucleation and
[217] I. Dutta, P. Kumar, G. Subbarayan, Microstructural coarsening in Sn-Ag-based growth, Trans. Am. Inst. Min. (Metall.) Eng. 135 (1939) 416e438.
solders and its effect on mechanical properties, JOM 61 (2009) 29e38. [253] M. Avrami, kinetics of phase change. I. General theory, J. Chem. Phys. 7
[218] Y. Sun, H. Fujii, Effect of abnormal grain growth on microstructure and (1939) 1103.
mechanical properties of friction stir welded SPCC steel plates, Mater. Sci. [254] B.L. DeCost, E.A. Holm, A computer vision for automated analysis and clas-
Eng. 694 (2017) 81e92. sification of microstructural image data, Comp. Mat. Sci. 110 (2015)
[219] H.V. Atkinson, Acta Metall. 36 (1988) 469e491. 126e133.
[220] D.A.H. Hanaor, W. Xu, M. Ferry, C.C. Sorrell, Abnormal grain growth of rutile [255] A. Chowdhury, E. Kautz, B. Yener, D. Lewis, Image driven machine learning
TiO2 induced by ZrSiO4, J. Cryst. Growth 359 (2012) 83e91. methods for microstructure recognition, Comp. Mat. Sci. 123 (2016)
[221] S.R. Kalidindi, Microstructure informatics, in: K. Rajan (Ed.), Informatics for 176e187.
Materials Science and Engineering, Elsevier, New York, 2013. [256] A.Y.M. Ontman, G.J. Shiflet, Application of artificial neural networks for
[222] S. Torquato, Random Heterogeneous Materials: Microstructure and Macro- feature recognition in image registration, J. Microsc. 246 (2012) 20e32.
scopic Properties, Springer, New York, 2002. [257] D.S. Bulgarevich, S. Tsukamoto, T. Kasuya, M. Demura, M. Watanabe, Pattern
[223] J.M. Rickman, K. Barmak, Kinetics of first-order phase transitions with recognition with machine learning on optical microscopy images of typical
510 J.M. Rickman et al. / Acta Materialia 168 (2019) 473e510

metallurgical microstructures, Sci. Rep.-UK 8 (2018) 2078. [274] M. Cragin, P. Heidorn, C.L. Palmer, L.C. Smith, An Educational Program on
[258] R. Liu, A. Kumar, Z. Chen, A. Agrawal, V. Sundararaghavan, A. Choudhary, Data Curation, ” ALA Science and Technology Conference Section, 2007.
A predictive machine learning approach for microstructure optimization and [275] A. Freitas, E. Curry, Big data curation, in: J. Cavanillas, E. Curry (Eds.), New
materials design, Sci. Rep.-UK 5 (2015) 11551. Horizons for a Data-Driven Economy, Springer, 2016.
[259] R. Kondo, S. Yamakawa, Y. Masuoka, S. Tajima, R. Asahi, Microstructure [276] H.M. Berman, et al., The Protein Data Bank, Nucleic Acids Res. 28 (2000)
recognition using convolutional neural networks for prediction of ionic 235e242.
conductivity in ceramics, Acta Mater. 141 (2017) 29e38. [277] N.B. Carlisle, J.M. Rickman, Workshop on the Convergence of Materials
[260] A. Cecen, H. Dai, Y.C. Yabansu, S. Kalidindi, L. Song, Materials structure- Research and Multi-Sensory Data Science: Roap, National Science Founda-
property linkages using three-dimensional convolutional neural networks, tion report, 2018.
Acta Mater. 146 (2018) 76e84. [278] A. Shaon, A. Woolf, An OAIS based approach to effective long-term digital
[261] G. Liu, L. Jia, B. Kong, S. Feng, H. Zhang, H. Zhang, Artificial neural network metadata curation, Comput. Inf. Sci. 1 (2008) 2e16.
application to microstructure design of Nb-Si alloy to improve ultimate [279] D.L. Poole, A.K. Mackworth, Artificial Intelligence: Foundations of Compu-
tensile strength, Mat. Sci. Eng. A 707 (2017) 452e458. tational Agents, Cambridge University Press, Cambridge, 2017.
[262] L. Exl, J. Fischbacher, A. Kovacs, H. Oezelt, M. Gusenbauer, K. Yokota, T. Shoji, [280] F.V. Jensen, Bayesian Networks and Decision Graphs, Springer-Verlag, New
G. Hrkac, T. Schrefl, Magnetic Microstructure Machine Learning Analysis. York, 2001.
arXiv:1808.03794v1. [281] M.R. Anderberg, Cluster Analysis for Applications, Academic Press, New York,
[263] It should be noted, however, that the promotion of abnormal grain growth is 1973.
sometimes desirable. For example, AGG is employed in so-called electrical [282] M.J. Zaki, W. Meira Jr., Data Mining and Analysis: Fundamental Concepts and
steels to improve their magnetic properties. Algorithms, Cambridge University Press, Cambridge, 2014.
[264] A. Lawrence, J.M. Rickman, M.P. Harmer, A.D. Rollett, Parsing abnormal grain [283] J. Patterson, A. Gibson, Deep Learning: A Practitioner's Approach, O'Reilly,
growth, Acta Mater. 103 (2016) 681e687. Boston, 2017.
[265] A. Necir, A. Rassoul, R. Zitikis, Estimating the conditional tail expectation in [284] J.A. Lee, M. Verleysen, Nonlinear Dimensionality Reduction, Springer, New
the case of heavy-tailed losses, J. Prob. Stat. 596839 (2010). York, 2007.
[266] J.M. Rickman, Y. Wang, A.D. Rollett, M.P. Harmer, C. Compson, Data analytics [285] O. Kramer, Genetic Algorithm Essentials, Springer, 2017.
using canonical correlation analysis and Monte Carlo simulation, J. npj [286] E. Alpaydin, Introduction to Machine Learning, MIT Press, Cambridge, 2004.
Comput. Mater. 3 (26) (2017). [287] K. Gurney, An Introduction to Neural Networks, Taylor and Francis, New
[267] J.D. Jobson, Applied Multivariate Data Analysis, vol. II, Springer-Verlag, 1992. York, 1997.
[268] S. Akaho, A Kernel Method for Canonical Correlation Analysis, International [288] I. Arel, D.C. Rose, T.R. Karnowski, “Deep machine learning e a new frontier in
Meeting of Psychometric Society, Osaka, Japan, 2001. artificial intelligence research”, IEEE Comput. Intell. Mag. 4 (2010) 13e18.
[269] S. Balakrishnan, K. Puniyani, J. Lafferty, Sparse additive functional and kernel [289] M.N. Murty, R. Raghava, Support Vector Machines and Perceptrons:
CCA, in: Proceedings of the 29th International Conference on Machine Learning, Optimization, Classification and Application to Social Networks,
Learning, 2012. Edinburgh, Scotland, UK. SpringerBriefs in Computer Science, 2016.
[270] D. Abou-Ras, N. Schafer, T. Rissom, M.N. Kelly, J. Haarstrich, C. Ronning, [290] I.T. Jolliffe, Principal Component Analysis, Springer, New York, 1986.
G.S. Rohrer, A.D. Rollett, Grain boundary character distribution and corre- [291] J.E. Jackson, A User's Guide to Principal Components, John Wiley and Sons,
lations with electrical and optoelectronic properties CuInSe2 thin films, Acta Hoboken, NJ, 2003.
Mater. 118 (2016) 244e252. [292] B. McCune, J.B. Grace, D.L. Urban, Analysis of Ecological Communities, MjM
[271] D.L. McDowell, S.R. Kalidindi, The materials innovation ecosystem: a key Software Design, 2002.
enabler for the Materials Genome Initiative, MRS Bull. 41 (2016) 326e335. [293] J.A.K. Suykens, J. Vandewalle, Least squares support vector machine classi-
[272] J.H. Panchal, S.R. Kalidindi, D.L. McDowell, Key computational modeling is- fiers, Neural Process. Lett. 9 (1999) 293e300.
sues in integrated computational materials engineering, Comp.-Aid. Design [294] E.R. Pachecho, Unsupervised Learning with R, Packt Publishing, Birmingham,
45 (2013) 4e25. UK, 2015.
[273] T. Kwartler, Text Mining in Practice with R, Wiley, Hoboken, NJ, 2017.