European Journal of Personality, Eur. J. Pers.

(2020)
Published online in Wiley Online Library (wileyonlinelibrary.com) DOI: 10.1002/per.2305

Using Big Data and Machine Learning in Personality Measurement:

Opportunities and Challenges

LEO ALEXANDER III*, EVAN MULFINGER and FREDERICK L. OSWALD

Department of Psychological Sciences, Rice University, Houston, TX USA
[email protected]

Abstract: This conceptual paper examines the promises and critical challenges posed by contemporary personality
measurement using big data. More specifically, the paper provides (i) an introduction to the type of technologies that
give rise to big data, (ii) an overview of how big data is used in personality research and how it might be used in the
future, (iii) a framework for approaching big data in personality science, (iv) an exploration of ideas that connect
psychometric reliability and validity, as well as principles of fairness and privacy, to measures of personality that
use big data, (v) a discussion emphasizing the importance of collaboration with other disciplines for personality psy-
chologists seeking to adopt big data methods, and finally, (vi) a list of practical considerations for researchers seeking
to move forward with big data personality measurement and research. It is expected that this paper will provide in-
sights, guidance, and inspiration that helps personality researchers navigate the challenges and opportunities posed
by using big data methods in personality measurement. © 2020 European Association of Personality Psychology

Key words: big data; machine learning; personality measurement

Today’s technologies, and the big data that they give rise to, data to support and improve future research and application
have dramatically infused our everyday lives and continue to efforts.
transform society in many domains, such as through social All these potential applications of big data for measuring
media, marketing, online education, and voting, and through and analysing personality and its outcomes pose new and
continued developments in areas such as autonomous interrelated technological, legal, and ethical questions and
vehicles, personalized medicine, and automation in the work- challenges for personality researchers and practitioners.
force. This ongoing transformation not only reflects the inex- Fortunately, the science of personality is ready to take on
orable impact of technology on society but also promises to this future, given its long and rich history of developing,
benefit science by revealing unique undiscovered aspects of implementing, and evaluating measures of personality. In this
individuals’ lives and personality. As researchers continue context, the current paper explores the research opportunities
to investigate personality within the arena of big data and ar- and challenges associated with big data personality measure-
tificial intelligence (AI), knowledge of an interrelated and ment, connecting future promise to acquired research wisdom
evolving system of benefits and challenges along ethical, le- by building on traditional psychometric concepts of reliability
gal, and scientific fronts is beginning to accrue. Furthermore, and validity, as well as principles of fairness and privacy.
as we continue to think about and investigate personality in Out of necessity, our overview will be general, given that
the domains of big data and machine learning, many ways personality research is just beginning to employ and investi-
of viewing big data personality measurement are likely to gate questions involving big data, AI, and machine learning,
be useful, for example, (i) as a cultural factor that allows a fast‐moving arena that opens many exciting doors for
socio‐technological contexts, relationships, and communica- personality research. First, we will introduce technologies
tions to develop in ways not possible just decades ago, creat- that have allowed for the collection of big data relevant to
ing new social behaviours and norms relevant to personality; personality. Second, we will provide a brief overview of
(ii) as an advisor and trainer that reciprocates between the small‐but‐growing body of existing big data personality
behaviour and adaptive self‐management interventions in a research. Third, we will offer an organizing framework for
real‐time system reflective of personality; (iii) as an analyser thinking about big data in personality science. Fourth, we
and informant that gathers, analyses, summarizes, and will re‐examine traditional psychometric evaluations of reli-
reports personality‐relevant information to decision makers ability and validity, as well as fairness and privacy, within
(e.g. teachers, supervisors, groups, and teams); and (iv) as a the modern context of big data algorithms as applied to
scientific resource that ideally provides personality‐relevant personality‐relevant big data. Fifth, we reflect on the critical
importance of multidisciplinary collaboration for personality
psychologists seeking to adopt big data methods. Finally, we
*Correspondence to: Leo Alexander III, Department of Psychological
Sciences, Rice University, 6100 Main St. MS25, Houston, TX 77005, USA.
will list key considerations that can inform future big data
E‐mail: [email protected] personality measurement and research.

Handling editor: John Rauthmann

Received 11 November 2019
© 2020 European Association of Personality Psychology Revised 25 August 2020, Accepted 25 August 2020
L. Alexander III et al.

DIGITAL TRACES OF BEHAVIOUR Wearable devices are particularly relevant to within‐

person variation in behaviour, as they are not only helpful
Modern digital technologies have given rise to many to a user’s understanding and self‐management, but they also
potential sources of personality‐relevant big data (see Woo, open up great possibilities for intervening in a more timely
Tay, Jebb, Ford, & Kern, 2020). Smartphones and other and tailored manner upon people at work, at school, or
personal electronic devices contain a variety of sensors in daily life (Ihsan & Furnham, 2018; Mardonova &
(e.g. microphones, cameras, light sensors, accelerometers, Choi, 2018). Personality may influence the type of interven-
and proximity sensors) and data logs (e.g. call logs, text mes- tion selected as well as how the intervention is responded to;
saging logs, web browser logs, and application use logs), and the interventions in turn may induce new habits, if not
both of which provide a rich source of longitudinal longer lasting personality change. For example, in the sports
behavioural data (Harari et al., 2016). Bluetooth wireless and medical domain, digital activity trackers (e.g. Fitbit and
data and GPS navigation features on cell phones can track Apple Watch) are being marketed to fitness enthusiasts as
where you are, when, and with whom—all of which may well as anyone seeking to improve their health (Aroganam,
be revealing of personality (Mønsted, Mollgaard, & Manivannan, & Harrison, 2019). The resultant fitness data
Mathiesen, 2018). Access to buildings, vehicles, and other may shed light, in aggregate groups and for individuals, on
secure locations increasingly rely on technologies that whether personality traits predict health decisions, processes,
record one’s specific whereabouts throughout the day (e.g. and outcomes such as (i) the choice to use these devices; (ii)
geographic location, building location via electronic access the nature and difficulty of self‐set health goals and; (iii) the
cards/codes, or video surveillance), which along with the as- process of how and when people revise, maintain, and attain
sociated time, and perhaps the people with which one was as- those goals effectively.
sociated (and their data), might yield personality‐relevant Among the many advances in digital technologies, social
information. Online downloads and cloud‐streaming services media platforms stand out as particularly rich sources of big
store data on whether you are listening to Mozart or data for personality researchers. Given the large number of
Metallica, which can be translated into musical preferences people who now regularly use social media networks
that, in turn, inform personality (Rentfrow & Gosling, 2003). (Lenhart et al., 2015; Perrin & Anderson, 2019), they often
Digital financial transactions such as online purchases, debit reflect not only extremely large samples and wide‐ranging
card, credit card, and payment app transactions continue to demographics (race/ethnicity, gender, age, geography, and
increase in popularity (Foster, Schuh, & Zhang, 2013) and culture) but also intensive longitudinal sampling, with digital
can be revealing of personality, as was demonstrated by trace data often accumulating multiple times per day. Perhaps
Gladstone, Matz, and Lemaire (2019), who used machine it is no wonder, then, as we will discuss later, much of the
learning techniques to predict personality from individuals’ existing big data personality research has relied upon data
bank records. Although the accuracy of their predictions supplied by social media platforms.
were small for the broad Big Five personality traits
(rs ¼ .12 to .19), they found larger correlations between their
model’s predictions and scores on survey‐based measures of THE CURRENT STATE OF PERSONALITY
the more narrow and substantively relevant traits of material- MEASUREMENT USING BIG DATA
ism and self‐control (rs ¼ .33 and .26, respectively).
Technological advances not only offer new sources of Initial research efforts using trace data for personality mea-
personality‐relevant data but also allow us to collect those surement were conducted primarily in the field of computer
data in new and more intensive ways. The trail for science, where predictive models were applied to big data
considering these data has been blazed by personality and to predict scores on measures of the Five Factor model of
methodological researchers who have focused strongly on personality (McCrae & John, 1992). This burgeoning body
investigating within‐person variability that is configural of research has identified many big data predictors of person-
(e.g. Molenaar & Campbell, 2009), context‐driven (e.g. ality as extracted from a variety of digital sources, such as in-
Fleeson, 2004), and time‐driven (Hamaker, Nesselroade, & formation from Facebook (Bachrach, Kosinski, Graepel,
Molenaar, 2007). We are thus witnessing a dramatic resur- Kohli, & Stillwell, 2012; Golbeck, Robles, & Turner, 2011;
gence of personality research focused on empirical tests of Schwartz et al., 2013; Sumner, Byers, & Shearing, 2011;
reliable patterns of within‐person variation that is either Wald, Khoshgoftaar, & Sumner, 2012; Youyou, Kosinski,
related to or directly reflective of different personality traits & Stillwell, 2015; Youyou, Stillwell, Schwartz, &
(see the review by Beck & Jackson, 2019a), for example, Kosinski, 2017), Twitter (e.g. Golbeck, Robles, Edmondson,
examining relatively fast change, using daily diaries in & Turner, 2011; Quercia, Kosinski, Stillwell, &
clinical assessment (Zimmermann et al., 2019); momentary Crowcroft, 2011; Sumner, Byers, Boochever, & Park, 2012),
assessments of personality‐performance relationships at Flickr (e.g. Yan et al., 2015), smartphone data (e.g.
work (Ilies & Judge, 2002; Judge, Simon, Hurst, & Chittaranjan, Blom, & Gatica‐Perez, 2013), and personal
Kelley, 2014; Minbashian, Wood, & Beckmann, 2010); lon- weblogs (e.g. Iacobelli, Gill, Nowson, & Oberlander, 2011).
ger term life‐event‐driven changes in personality (Bleidorn, Researchers continue to explore a variety of trace data from
Hopwood, & Lucas, 2018); and developmental personality these and many other sources, including social media profile
changes in making school‐to‐work transitions (Lüdtke, information, images, various user logs, messages, posts, and
Roberts, Trautwein, & Nagy, 2011). Facebook and Twitter ‘likes’.

© 2020 European Association of Personality Psychology Eur. J. Pers. (2020)

DOI: 10.1002/per
 Using Big Data and Machine Learning in Personality Measurement: Opportunities and Challenges

For example, the myPersonality project (Stillwell & do not focus on personality or any psychological characteris-
Kosinski, 2020) developed an app where, until it closed in tics but instead are broadly sensitive to unique characteristics
2012, an astonishing six million Facebook users had taken at of the data arising from the aforementioned digital technolo-
least one of a suite of online psychological tests available, gies, quite often employing several Vs in doing so, for exam-
including self‐report and peer‐report Big Five personality ple, the historically well‐known volume, variety, and veracity
measures. Of those respondents, 40% agreed to share their of big data proposed nearly 20 years ago (Laney, 2001); but
anonymized data with the myPersonality project. Until the au- also visualization, variability, vulnerability, visibility, vague-
thors stopped sharing myPersonality data in 2018, these data ness—and still other Vs (Storey & Song, 2017). Such defini-
were made available to other researchers, leading to dozens tional terms are important for their useful abstractions, yet as
of peer‐reviewed journal publications and greatly expanding any personality researcher dealing with big data would tell
the body of big data personality research (see mypersonality. you, the devil of understanding big data lies in the important
org for a reflection on the project and a selected bibliography). details associated with any particular application and setting.
In addition to such large‐scale studies, two recent meta‐ For example, technology interacts with humans, in addition
analyses have been conducted to estimate correlations be- to recording their behaviour (e.g. multimedia platforms use
tween social media digital trace data and traditional Big Five algorithms to suggest media as well as record a user’s prefer-
questionnaires (Azucar, Marengo, & Settanni, 2018; Settanni, ences in media), and thus, the influences of AI technology
Azucar, & Marengo, 2018). Meta‐analytic estimates suggest and human personality will be reciprocal. We say this know-
that correlations between digital trace data and measures of ing that technologies will always keep improving in ways
personality range from .29 to .40 across the Big Five person- that will better evoke, detect, and reflect personality (e.g.
ality traits, perhaps a reasonable level of convergence given more effective AI, new data analytic techniques, faster
the noisiness of trace data. Comparing these meta‐analytic processing speed, and greater storage capacity; Gandomi &
findings with a meta‐analysis of the accuracy of human ratings Haider, 2015).
of personality based on social media data (Tskhay & Rule, In the context of these changes, our perspective on big
2014), computer algorithms were at least more consistent than data relevant to personality research leans towards the practi-
human raters (Youyou et al., 2015), if not more accurate given cal: definitions and frameworks for big data are useful not
the absence of a gold standard for measures of personality. only for their content but also as a tool for cultivating a
Although using machine learning and big data to predict growing community of psychological and interdisciplinary
scores on the Big Five is useful, there is a much more exten- researchers who engage in a wide range of projects and ap-
sive and continuing need to understand convergent and dis- plications involving personality‐oriented big data. Keeping
criminant patterns involving machine learning models and this perspective firmly in mind, we selected the framework
personality‐relevant data within a much larger nomological proposed by De Mauro, Greco, and Grimaldi (2015) as a
net of psychological constructs (Campbell & Fiske, 1959). general and practical guiding framework for approaching
Related to this need, additional research could also pursue big data (versus traditional data sets). These authors surveyed
three key questions about big data: (i) whether they yield a wide range of definitions of big data and identified four
unique personality‐relevant variance; (ii) whether they pro- common and distinct themes.
vide incremental prediction of school, organizational, and
life outcomes above traditional Big Five measures (Roberts,
Information
Kuncel, Shiner, Caspi, & Goldberg, 2007); and (iii) whether
this incremental prediction can be attributed to personality Most frequently, definitions of big data include information or
and/or to other constructs. the size and structure of the data itself. This is related to the
As we have briefly described, research has already begun aforementioned ‘Vs’ and it is worth pointing out that in the
to make concerted efforts in predicting personality from big history of psychological science, Cattell’s data box (Cattell,
data; but whether additional personality insights actually 1946) parallels this information component of big data
result from big data seems to remain fertile investigative soil, (Adjerid & Kelley, 2018). The data box is conceptual and
for example, how have the clustering and/or predictive reflects three dimensions of information—people × variables,
algorithms involving big data improved our understanding people × occasions, and variables × occasions—where diverse
of personality? And how does an improved understanding types of rich within‐person and between‐person big data can
of personality translate into people’s functioning and populate this matrix (e.g. experience sampling methodologies
well‐being in their daily lives at work, school, and home? can vary in their sampling rates in terms of people, variables,
Before we partially address these questions in the current pa- and occasions; Haqiqatkhah & Tuerlinckx, 2019). Also histor-
per, let us first place some useful contours around the mean- ically, Cronbach and colleagues’ generalizability theory is an
ing of big data in personality research. important‐yet‐underappreciated extension of the data box
(Cronbach, Gleser, Nanda, & Rajaratnam, 1972; Cronbach,
Rajaratnam, & Gleser, 1963) to estimate the variance along
A FRAMEWORK FOR BIG DATA IN THE an infinite variety of dimensions (known as facets in general-
PERSONALITY CONTEXT izability theory), such as units, treatments, operations, and
settings, called UTOS (Cronbach, 1982). The collection, anal-
So many definitions of big data exist that they could them- ysis, and interpretation of personality‐oriented big data can
selves populate a big data set. Typically, these definitions usefully benefit from this historical foundation of thinking

© 2020 European Association of Personality Psychology Eur. J. Pers. (2020)

DOI: 10.1002/per
L. Alexander III et al.

that comes with Cattell’s data box and Cronbach et al.’s of these and other popular big data frameworks, see
generalizability theory, because it lays out an important Inoubli, Aridhi, Mezni, Maddouri, & Nguifo, 2018; also
framework for research projects, programmes, and related see mobileQ, https://mobileq.org/, Meers, Dejonckheere,
efforts pertaining to replication and meta‐analysis (Nosek & Kalokerinos, Rummens, & Kuppens, 2019). Interactions in
Errington, 2020; Simons, Shoda, & Lindsay, 2017). In this games, immersive virtual reality environments, and conver-
context, an important dimension of personality measurement sations with AI avatars are just a few examples of modern
from big data is methodology, because the same set of big data technologies that may provide big data on personality.
might be analysed with any of a wide range of machine learn- Certainly, modern digital technologies will affect the na-
ing algorithms (e.g. random forests, support vector machines, ture of personality‐relevant big data and their subsequent
and elastic net regression). Extending Cronbach’s term to in- analysis in many ways, both intended and unintended. For
clude methodology, UTOS then becomes MUTOS (Becker example, video surveillance of employees may provide reli-
& Aloe, 2008). able data and analysis pertaining to customer service, in
Another important parallel with the methodological terms of sociability, conscientiousness, conflict avoidance,
traditions in psychological research comes with the big data and so on. However, much like the classic Hawthorne studies
Vs of veracity and value, which respectively tie into the (Roethlisberger & Dickson, 1939), when employees know
well‐established concepts of reliability and validity found they are being continuously surveilled, the implementation
in psychometrics, as well as the concept of utility found in of data‐gathering and automated technologies in the work-
industrial‐organizational psychology. Veracity refers to the place may yield unintended negative effects on employees
quality and accuracy of big data, which is often lower for in- responses/behaviours (e.g. approaching customers more than
dividual data points that are seemingly related to personality is necessary), and reactions will likely vary depending on
(e.g. a single Facebook ‘like’), but can dramatically personal characteristics (Yost, Behrend, Howardson, Darrow,
increase when appropriately aggregated across the multiple & Jensen, 2019) and demographic characteristics (e.g. age,
personality‐related situations and time points that big data gender, race/ethnicity, and socio‐economic status). If one
offer. Note that this point parallels the historical discovery seeks to understand the culture of modern work, education,
of psychometrically reliable traits via the aggregation of and home settings involving big data, then some form of per-
personality‐relevant behaviours (Epstein, 1983). In the big sonality measurement must be an essential part of it.
data realm, one may aggregate big data by variables (e.g. This raises an important concern surrounding the objec-
grouping the columns of a data set via principal components tivity of big data measures of personality, which has obvi-
analysis or factor analysis), by subjects (e.g. grouping the ously been a longstanding concern with traditional
rows of a data set via k‐means clustering or hierarchical ag- personality measures. Psychometrically, traditional personal-
glomerative clustering), or by variables and subjects jointly, ity tests attempt to minimize variance in construct‐irrelevant
such that personality‐based signals can be built up and factors, meaning that test scores are as unaffected as possible
contrasted against other systematic effects and errors found by both systematic and random errors related to test adminis-
in a vast sea (or messy matrix) of big data. tration, content, and test responding. By contrast, just as re-
Value is a big data term that often refers to validity and/or searchers hope to find personality‐relevant variance in big
utility, for example, personality‐relevant big data can hold data, there is also a massive suite of systematic and random
value to the extent that it predicts meaningful outcomes in irrelevancies that an algorithm can be trained on, leading to
academic, organizational, and other developmental, health, confounds we tend to call algorithmic bias (Barocas &
and life domains (i.e. validity) and/or for the fact that those Selbst, 2016). At best, these irrelevancies will add random
predictions usefully inform personal or institutional goals errors to the measurement of personality using big data (the
(i.e. utility). Thus, the value in big data personality research rough equivalent of lowering Cronbach’s alpha). At worst,
is often dependent on one’s perspectives and goals. Organi- incorrect conclusions and decisions may result if such irrele-
zations might define and quantify value and utility as a finan- vancies vary systematically with participant characteristics
cial return on investment (Cascio, Boudreau, & Fink, 2019). (e.g. demographics), individual differences (e.g. personality
But for researchers, big data findings may hold value to the itself, cognitive ability, and video game experience), and sit-
extent that they can contribute to traditional ways of accumu- uational influences (e.g. Internet bandwidth, computer equip-
lating knowledge about effect size, predictions, and their ment). To provide an example that raises these issues,
generalization, as learned from patterns of relationship over previous research has found that slightly more women,
time (Cronbach & Meehl, 1955), and via natural and formal African Americans, Hispanics, and younger applicants used
interventions (Chester & Lasko, in press). mobile devices to complete high‐stakes employment assess-
ments than other demographic groups (Arthur, Doverspike,
Muñoz, Taylor, & Carr, 2014), and it is expected that those
Technology
devices generally will not impair the application process
Many big data definitions also emphasize the specific and relative to other application methods. Although mobile
often‐sophisticated hardware and software technologies that technologies tend to show small or no subgroup differences
allow for the intensive real‐time and large‐scale collection for traditional personality measures (Arthur, Keiser, &
and management of personality‐relevant data. In addition to Doverspike, 2018), one should be sensitive to potential
commercial efforts, open‐source frameworks such as Hadoop subgroup differences in big data, personality‐oriented and
and Spark have been developed for this purpose (for a survey otherwise, as there are then potential fairness and legal

© 2020 European Association of Personality Psychology Eur. J. Pers. (2020)

DOI: 10.1002/per
 Using Big Data and Machine Learning in Personality Measurement: Opportunities and Challenges

implications in terms of ensuring the data are based on the dynamic/streaming framework (e.g. Ippel, Kaptein, &
equal access and use of technology. Vermunt, 2019). In other words, big data may be analysed
Overall, it cannot be assumed wholesale that big data are in myriad ways, and these choices have important implica-
personality relevant or can reliably predict more established tions for how and even whether research questions about per-
measures of personality constructs. Historically, traditional sonality can be answered with the available data.
computerized and pencil‐and‐paper versions of personality While keeping this last point in mind, we will now focus
inventories have been closely examined psychometrically on machine learning, which people often speak of as if it
in many ways to determine the strength of construct measure- were a monolithic entity, when as noted previously, it
ment and the nature and extent of systematic construct actually comprises a huge array of analytic techniques.
irrelevancies (e.g. because of measurement method, Appreciating that no single machine learning algorithm can
item‐specific content; Arthur, Glaze, Villado, & Taylor, 2010; ever be considered uniformly superior across all contexts
Meade, Michels, & Lautenschlager, 2007; Salgado & (Domingos, 2012), as well as appreciating the lack of guid-
Moscoso, 2003). By contrast, most sources of big data reflect ance in personality research for selecting among machine
a nature or purpose other than personality measurement and learning algorithms, we can at least present how machine
are often used for prediction but with very little consideration learning algorithms are organized into two broad and distinct
for construct measurement. Big data would therefore benefit categories: clustering and predictive modelling.
from psychometric modelling and methods (even if these Clustering is similar in spirit to traditional methods of
models/methods depart from traditional methods) to help en- profile analysis, where cases (rows) can be usefully summa-
sure that personality construct variance is being measured, rized by clusters that are internally consistent yet externally
while minimizing or avoiding deleterious contextual effects distinct. Popular clustering methods include k‐means, hierar-
(e.g. distortions and biases). And likewise, more traditional chical agglomerative clustering, and spectral clustering. Like
and psychometrically well‐developed personality measures profile analysis or factor analysis, clustering methods have
can contribute to and be compared with more organic forms no external criterion for informing each case’s probability
of big data. of cluster membership, which is why clustering is also called
Technology can influence more nuanced and less obvious unsupervised learning in the big data literature.
characteristics of personality measurement with big data as In a sense, the central aim of clustering is to group
well. For example, perhaps theory and thinking suggest that cases or rows of a data set such that the number of
a particular measurement should be captured at certain inter- clusters/groups maximizes the between‐cluster versus
vals (e.g. every 10 seconds), but the data were collected by a within‐cluster variance (in the spirit of ANOVA). Most
device that recorded measurements at longer intervals (e.g. clustering methods assign each case a probability of mem-
30 seconds) or perhaps were event‐driven instead (e.g. the bership to each cluster, where each case ‘belongs’ to the
data were captured whenever certain patterns of movement, cluster with the highest associated probability. In the per-
physiology, physiognomy, or location are exhibited). sonality context, clusters might reflect profiles of traits that
Although these measurement considerations generally and are informative and provide information beyond each trait
typically exist in all research design, the point here is to ap- considered as a main effect. For some time, there has been
preciate how the choice and design of technology interacts a strong and longstanding debate that suggests that typolog-
with conceptual considerations, practical constraints, and ical approaches (idiographic, or profile‐based, cluster‐based,
possible analyses. or type‐based approaches) to personality are generally
not robust predictively or parsimonious conceptually
(Donnellan & Robins, 2010; however, see the aforemen-
Algorithms
tioned Molenaar & Campbell, 2009). But perhaps machine
Regarding this last point, the digital technologies that yield learning algorithms applied to big data will change that,
big data may also influence what analytical techniques are bringing new insights and additional value to the typologi-
appropriate (and inappropriate) through the types of the data cal approach: for recent inspiration, see the four personality
that they produce. For example, when personality‐relevant types identified using machine learning techniques applied
big data can be collected, integrated, and analysed simulta- to a Big Five measure in Gerlach, Farb, Revelle, and
neously on a real‐time or streaming basis from multiple tech- Amaral (2018) and an important rebuttal by Ones and
nological sources (e.g. video, cell phone, and social media Wiernik (2018).
posts), the modelling of the data (and also the prior cleaning Predictive modelling is similar to regression analysis,
and preparation of the data) may necessarily become com- where, unlike clustering, a dependent variable (or variables)
plex along multiple levels. For instance, imagine personality is predicted from a set of continuous and/or categorical pre-
big data, collected over time, dynamically predicting out- dictors (often referred to as features in predictive modelling).
comes for students (i.e. engagement, teamwork, and grades), Having a criterion is the hallmark of predictive modelling
teachers (i.e. giving timely and supportive feedback and pro- with supervised learning algorithms. As of this writing, the
viding accurate information), and institutions (i.e. overall caret package within the R programming language contains
student retention and graduation rates). This type of complex 238 machine learning algorithms for prediction, a huge num-
prediction problem could be modelled with any of ber that only keeps growing (see https://topepo.github.io/
hundreds of types of machine learning algorithms, or it could caret/available-models.html; Kuhn, 2019). Although the
be modelled explicitly in a multilevel, structural, and algorithms differ widely, they share the central goal of

© 2020 European Association of Personality Psychology Eur. J. Pers. (2020)

DOI: 10.1002/per
L. Alexander III et al.

achieving robust prediction by doing two things not typically where one must decide whether to gain some predictive
found in traditional statistical analyses of personality data: power at the expense of a more straightforward and interpret-
able model (or vice versa).
1 Disturb the model. Certain machine learning algorithms
Given this challenge of interpretability, why should one
tend to combine predictive results across hundreds of
even consider using machine learning algorithms in person-
weak models to result in a stronger prediction (e.g.
ality measurement with big data? There are two very practi-
averaging across hundreds of trees in a random forest).
cal reasons worth emphasizing: (i) the number of variables
Or in a similar vein, other algorithms might examine a grid
exceeds the number of cases in a data set (e.g. because text,
of possible model parameters (hyperparameters) to
audio, and social media data sets are vast), meaning that tra-
‘tune’ the model in the search for complex yet
ditional analyses such as multiple linear regression are im-
robust relationships (e.g. the learning rate parameter in
possible (i.e. the matrix will not be invertible; Fan &
gradient‐boosted machines; the cost parameter in support
Li, 2006), and (ii) the researcher seeks to go beyond tradi-
vector machines).
tional analyses, to see whether robust complex relationships
2 Disturb the data. Virtually all machine learning models
(ones that may not be specified a priori) can be located,
are concerned with some form of cross‐validation that
and prediction can improved without overfitting the data
keeps the data for model development and the data for pre-
(i.e. mistaking sampling error variance for actual complexity;
diction separate. In 10‐fold cross‐validation, for instance,
Hawkins, 2004).
the data set is randomly divided into 10 sections or folds;
To move beyond these general considerations and learn
9/10ths of the data is considered the training set, used to
more about what machine learning is and how to apply it,
develop the model; and then predicted values generated
most personality psychologists would benefit from the
for the test set, or the 1/10th of the data that did not partic-
practical guide to big data in psychology by Chen and
ipate in model development. This process is repeated nine
Wojcik (2016), as well as the psychology‐oriented big data
more times so that every case gets to be in the test set with
tutorials pertaining to text mining or web scraping (e.g.
a predicted value. Bootstrapping methods contain another
Landers, Brusso, Cavanaugh, & Collmus, 2016), data
form of cross‐validation, where a model is first estimated
processing and predicting outcomes (e.g. Kosinski, Wang,
from the bootstrap sample (subsampling with replace-
Lakkaraju, & Leskovec, 2016), and meta‐analysis (e.g.
ment); then predictions are made for those cases from
Cheung & Jak, 2016).
those data that did not participate in a given model; and
this process is repeated a large number of times.
By disturbing both the model and the data, the wide range
Impact
of machine learning techniques attempt to seek out the best
model among those searched (#1 above) in terms of success- The last major theme contained in many definitions of big
ful predictions (e.g. lowest mean‐squared error) under data reflects the impact of big data, or how the output from
cross‐validation (#2 above). big data algorithms is used as feedback for our behaviours:
In addition to clustering versus predictive modelling, ma- informing, interacting with, and potentially magnifying, mit-
chine learning algorithms can also be considered in terms of igating, or otherwise influencing them. These reciprocal ef-
their interpretability (Ribeiro, Singh, & Guestrin, 2016a), fects between algorithms and behaviours might only be
which has obvious important ties to scientific understanding, understood within short time spans, although it is possible
policymaking, fairness, ethics, and efforts to improve both that habits, personality traits, and behaviours could be af-
data and algorithms. Some algorithms are more interpretable fected in the longer term as well. This type of iteration and
than others. For example, it is relatively straightforward to impact should be considered in the context of the range of
explain the personality profiles of cluster means (centroids) settings in which big data are used, from the settings that
in k‐means clustering or k‐nearest‐neighbour prediction, or might be relevant at the individual level, to the broad and ag-
to explain the regression coefficients of personality predic- gregated settings of big data analysis that might be relevant at
tors found in lasso or elastic net regression. In these cases, the community and policy levels. For example, if we con-
the models and processes by which these algorithms sider how a user’s personality affects how and when they fol-
evaluate the associations between inputs and outputs are low up on recommendations from a smartphone health app,
relatively transparent. By contrast, there are many black‐ then the individual‐level impact on health, privacy, and
box algorithms that, as the name suggests, are relatively safety over time may be of paramount importance. However,
opaque. Often the complexities explored by these algorith- for scholars and policy makers who are focused on how per-
mic techniques do not have a closed‐form function and are sonality data from social media and résumé websites are used
therefore much more difficult, if not impossible, to interpret to recruit and hire employees, then the societal‐level focus
(e.g. artificial neural networks tune network weights within may be on trends in national productivity, worker rights, or
arbitrary layers of hidden nodes; random forests average legal liabilities. Applications in all these settings, and at all
across hundreds of trees of variables; support vector ma- these levels, have ethical challenges for applying knowledge
chines use nonlinear profile matching along classification concerning personality, where enumerating, discussing, and
and prediction boundaries; see James, Witten, Hastie, & weighing the pros and cons often do not lead to obvious or
Tibshirani, 2013). Explanation versus prediction often (but single solutions and therefore must be explored deeply and
not always) reflects a trade‐off found in machine learning, on a continuous basis.

© 2020 European Association of Personality Psychology Eur. J. Pers. (2020)

DOI: 10.1002/per
 Using Big Data and Machine Learning in Personality Measurement: Opportunities and Challenges

The potential impact of big data that is perhaps most rel- algorithms, and impact. Thus, the reader should appreciate
evant to personality researchers is how these data might in- that the value of personality measurement using big data de-
crease the scientific understanding of personality. For pends on the combination of these categories—none of these
example, large behavioural data streams may identify new should be considered in isolation. For example, the technol-
indicators and aspects of personality that have not previously ogy used to collect data (e.g. a social media platform) will
been revealed via traditional personality measures, such as determine the types of personality relevant big data available
Big Five measures based in the lexical tradition (Bleidorn, (e.g. large and sparse data sets of social media ’likes’ where
Hopwood, & Wright, 2017). Moreover, novel types of be- there are more variables than cases). Consequently, these
havioural variables found in digital trace data may advance types of data may be more readily analysed by a certain algo-
personality research through revealing new insights about rithm or set of algorithms (e.g. using clustering algorithms to
trait content and trait‐related processes. Big data insights reduce the dimensionality of the data, and then predictive al-
such as these can lead to developing measures of new or ex- gorithms to classify the media platform’s users based on
panded constructs in the future, ones that may amplify the some dimension of personality). In turn, the algorithmic ap-
original big data signal over the noise even further. For ex- proach chosen may be more or less interpretable than other
ample, the lexical hypothesis, upon which many Big Five approaches, having implications for the impact of using ma-
measures of personality have been developed, assumes that chine learning algorithms in situations where a clear under-
all personality can be captured in our language (e.g. intro- standing of what has driven a prediction is useful or
spective or retrospective self‐report questionnaires). How- necessary, such as when the use of big data may or may
ever, if some aspects of personality are better captured not be legal (e.g. employee recruitment) or safe (e.g. medical
through behaviour, then perhaps existing personality con- treatment recommendations). A good practice when planning
structs will be refined further, or new personality constructs big data personality analyses is to generate options within
will emerge from vast streams of personality‐relevant behav- these four categories, appreciating how the choices made in
ioural data. Also, given the intensive sampling that big data one category often influence the options available in another
offer, new patterns of personality change may be discovered (e.g. the choice of technology used to collect big data influ-
that are reliable and predictable, if not dispositional. ences the information available). Mapping out multiple sce-
Other aspects of big data may shed light on personality narios in this way, and comparing them with one another,
development and the expression of personality in various can also be helpful in this planning process.
contexts. The ubiquity of big data, the large diverse samples
they offer, and their temporal granularity promise to yield
greater insight into the reliability of personality indicators BIG DATA CHALLENGES TO RELIABILITY,
over one’s lifespan and across cultures (see Bleidorn VALIDITY, FAIRNESS, AND PRIVACY
et al., 2017). Additionally, big data and machine learning
might complement current advances in personality research, Big data involving personality‐relevant information, accom-
for example, through measuring and analysing massive panied by robust predictions, have the potential to improve
amounts of data collected across the lifespan of huge samples our understanding of personality. In fact, the reason re-
that traditional technologies and statistical methods simply searchers and practitioners are investing in big data and ma-
cannot. Through the richness of big data, we might gain more chine learning is to obtain new and better personality
ecologically valid insights into personality development (and insights, obtained through more sophisticated‐yet‐robust re-
stability) across the lifespan, and how personality facilitates, lationships, and a better overall understanding of personality
limits, or otherwise interacts with situational factors. For big constructs. However, as exciting as this might sound, bene-
data that are multicultural and international in nature, this big fits do not accrue automatically by bringing big data and ma-
data approach promises to help researchers gain a deeper un- chine learning together. Instead, in line with the
derstanding of the culture‐free (etic) and culture‐specific hypothetico‐deductive cycle, personality researchers must
(emic) components of personality, and not merely enhance continue to iterate between top‐down intentional forms of
predictive validity (although these are often interrelated big data (those resulting from measures based in personality
goals). As a prime example, the large, culturally diverse data theory) that inform and are informed by bottom‐up incidental
sets offered by the many digital communication platforms big data (those that happen to be available). Such an iterative
used throughout the world provide a wealth of personality process of empirical analysis motivating the refinement and
data where researchers can compare the quality, reliability, revision of psychological theory (and vice versa) has a long
and validity of big data indicators of personality in many dif- scientific history (Fiedler, 2018). This process will continue
ferent populations, in many different settings, and over long with big data analyses as it has with traditional data analyses,
periods of time. where, both within and across research endeavours, we con-
tinuously accrue information about big data reliability and
validity that informs personality theory.
Applying this framework to big data personality
measurement
Reliability
As our discussion of this framework suggests, the measure-
ment of personality using big data is complex and involves The nature of big data poses many challenges to quantifying
multiple interrelated categories: information, technology, sources of construct‐relevant variance and error variance,

© 2020 European Association of Personality Psychology Eur. J. Pers. (2020)

DOI: 10.1002/per
L. Alexander III et al.

and how to create an appropriate reliability model in doing We have discussed just a few of the implications the in-
so. Using the framework, which we previously introduced, formation and technology aspects of big data may have for
it is likely that appropriate reliability estimates will be reliability, but the complexity of the analyses allowed for
heavily influenced by the unique combination of informa- by algorithms themselves also pose challenges to under-
tion, technology, and algorithms within each study or standing the reliability of personality measurement with big
application. The complexity and messiness of big data—par- data. In a recent example, machine learning algorithms have
ticularly when the number of variables exceeds the sample proven useful in attempts to analyse idiographic or
size—will inevitably demand novel approaches to estimating within‐person models of personality, meaning that each per-
reliability. Perhaps in some cases, traditional concepts of son contains variation and covariation in their own trait ex-
reliability can still be of use (e.g. internal consistency, test– pressions that can be modelled over time. Beck and
retest, parallel forms, and interrater agreement), but we sug- Jackson (2019b) used experience sampling method (ESM;
gest that new conceptions of reliability, or hybrid estimates Csikszentmihalyi & Larson, 2014) data to model the
that combine traditional measures of personality with big time‐lagged and contemporaneous relationships between
data, might find new ground and new importance in the con- within‐person manifestations of personality using the inter-
text of big data. pretable machine learning method of the graphical lasso
For example, a big data set may contain movie ratings, (Friedman, Hastie, & Tibshirani, 2008; an extension of
but the matrix is sparse because individual subjects view Tibshirani, 1996). The authors found contemporary idio-
only a very small subset of those movies. Therefore, reli- graphic models to be relatively consistent over time, whereas
ability and agreement are challenging to estimate (e.g. lagged models were not; however, both models were found
rank‐order stability, absolute accuracy, and factor structure). to demonstrate individual differences in consistency over
To add even more complexity to this modelling problem, time. Specifically, the results indicated that within‐person
ratings might be time dependent (e.g. holiday movies are networks of personality variables were relatively consistent
rated higher over the holiday of interest); person dependent with those measured in a second wave ESM study conducted
(e.g. only certain people celebrate certain holidays); and a year later when the personality variables were measured at
people drop in and out of the data set arbitrarily. These the same time points (i.e. contemporaneous). However, when
and other factors may have independent and joint effects ESM measurements were lagged by about 4 hours (to model
on big data as a reflection of personality. Here, various the daily dynamics of personality variables), the models were
forms of network analyses (Epskamp, Rhemtulla, & not consistent with those measured in the second wave of the
Borsboom, 2017) may take the place of traditional reliabil- study. Although this represents a very novel research applica-
ity estimates (see Christensen, Golino, & Silvia, 2020, and tion of big data, it is interesting to consider how machine
Costantini et al., 2019, for inspiration specific to personality learning models might begin to shed light on scientific ques-
research). tions that involve the prediction of dynamic within‐person
Thus far, we have only discussed the reliability of person- personality processes unfolding over time and individual dif-
ality big data in relation to the characteristics of the data ferences in their reliability.
themselves, but the technology used to collect the data also To move research of this nature forward even further,
carries important implications for reliability. Returning to others have argued for understanding the psychometric prop-
our previous example, not all movie raters in our hypotheti- erties of experience sampling personality state measures
cal big data set are watching movies on the same device or (Beckmann & Wood, 2017), an important application of the
in the same environment. Some subjects in the data set may need for personality researchers to improve their understand-
have rated a movie after watching it on a 60‐inch smart TV ing of the reliability of big data. Although much research has
with a fast fibre optic Internet connection in their spacious been conducted on big data indicators of personality (see the
home theatre, whereas others watched the same movie on meta‐analysis by Azucar et al., 2018, for a list of studies
the 5‐inch screen of their iPhone with a sporadic Internet predicting Big Five personality traits from digital footprints
connection on the subway train during their commute home on social media), most of this research has been focused
from work. This begs the empirical question, how reliable on the prediction of traditional measures of personality
are such ratings across different devices and different (Wright, 2014), such that outside of this sphere, much about
settings? At this point, we cannot answer this question in the reliability and other psychometric properties of big data
too much detail. However, it seems clear that understanding indicators of personality have yet to be evaluated. Reliability
systematic variation in the reliability of big data indicators can take many forms that are analogous to traditional ones,
of personality across such technological contexts would even if it is modelled differently, for example, internal con-
serve a very practical purpose that informs appropriate data sistency, longitudinal, and alternate forms. Psychometric
collection and interpretation. For example, time‐intensive modelling of big data will be an important direction, because
data‐collection technologies, along with the types of data that reliability is an important feature of all data, yet as we have
are collected, might affect the intervals at which we should already emphasized, personality‐relevant big data are often
assess (and re‐assess) the reliability of big data personality messier (less structured, more variables) than the personality
measures. Importantly, the interest in personality‐related big data and items coming from traditional psychological tests.
data opens the door to understanding real‐time changes in In short, reliability is just as important for understanding
data quality, reliability, validity, and subgroup differences— personality in the big data era as it has always been, because
and how they are related. there are scientific, practical, and ethical imperatives to

© 2020 European Association of Personality Psychology Eur. J. Pers. (2020)

DOI: 10.1002/per
 Using Big Data and Machine Learning in Personality Measurement: Opportunities and Challenges

separating variance relevant to constructs (e.g. personality, Of course, up until this point, our discussion of construct
motivation, knowledge, and otherwise) from variance irrele- validation has focused on identifying big data indicators that
vant to constructs (e.g. demographics, rater biases, reflect the Big Five and other established personality con-
item‐specific variance, and otherwise). We thus look forward structs. Beyond this application, however, there lies the idea,
to seeing personality psychologists informing data scientists the potential, for advancing personality theory by introduc-
(e.g. applied statisticians and computer scientists), and vice ing novel unique personality predictors that may reflect, ex-
versa, when developing new approaches to evaluating the re- tend, or even replace existing theories. For instance,
liability of personality‐relevant variance within big data sets. conscientiousness is viewed as a personality trait, yet perhaps
new technologies for measuring conscientiousness help to re-
define the construct by measuring how it changes in its na-
ture over time and how it predicts behaviour over time (e.g.
Novel measurement approaches
as a process in school, Corker, Oswald, & Donnellan, 2012).
In traditional measurement, the purpose of reliability as a part On the other hand, there is also the strong caution and poten-
of construct validation has been to provide evidence that the tial danger for complex predictive algorithms applied to
observed variance in scores on a measure of personality is re- messy big data sets to contribute to the jingle‐jangle person-
lated to the underlying latent personality construct being ality constructs (Kelley, 1927; Thorndike, 1904), essentially
measured (Borsboom, Mellenbergh, & Van Heerden, 2004). by empirically repackaging and combining data (new wine)
This task can be extremely difficult for big data, because re- that may reflect established personality constructs (old
searchers and respondents have less control over how the bottles). In considering these promises and perils for novel
data are produced. For example, the very same technologies construct definitions and measurement, note that the mas-
and predictive algorithms that collect big data in the first siveness of big data combined with the opacity of machine
place (e.g. Facebook likes, Twitter posts, Netflix movie rat- learning could lead to innovative and personality‐relevant
ings, and Amazon product ratings) are being used to select variance going unnoticed if we are not careful. To avoid
and tailor interventions that inform, amplify, reduce, or other- such confusion, it will be illuminating for research to con-
wise condition behaviour (e.g. future consumer purchases). tinue to compare and contrast personality constructs
Turning back to our movie example, a person’s movie (established versus novel), personality measures (traditional
choices and ratings lead to movie recommendations—and Likert scales versus adaptive item‐response‐theory‐based
an algorithm’s recommendations are based on the continuous tests versus big‐data‐based), and analysis tools [exploratory
analysis of viewer choices and ratings. The general point factor analysis versus confirmatory factor analysis (CFA)
here is whether algorithmic recommendations based on for clustering; linear regression versus structural equation
personality‐related behaviour are premature, just right, too modeling (SEM) versus machinelearning algorithms for
late, or something else. In this way, personality is in a net- prediction].
work of dynamic relationships with behaviour and the inter-
ventions received. This design is by intent; however, it makes
The practical and scientific value of big data in
the behavioural big data harder to interpret as purely trait
personality measurement
based (Kosinski, Matz, Gosling, Popov, & Stillwell, 2015).
Therefore, anchoring our predictive modelling decisions For many, the most exciting aspect of big data in personality
in theory and remaining mindful of possible confounds, measurement lies in its many real‐world applications. Inter-
while examining the content validity of big data personality active technologies allow for targeted intervention on human
indicators of personality, is our first line of defence against emotion, thought, and behaviour, and this dynamic data, in
training our models on such construct irrelevancies. Cer- turn, can be related to personality, where a large body of
tainly, well‐trained expert raters can attempt theoretical justi- research has demonstrated relationships between personality
fication for the inclusion or exclusion of potential big data and health and well‐being (Strickhouser, Zell, &
personality indicators, but they are inevitably limited to the Krizan, 2017). Research has also found relationships be-
potential indicators present in the data set, which might con- tween personality and compliance to medical treatment
tain both construct irrelevancies and construct deficiencies. (Umaki, Umaki, & Cobb, 2012) as well as prescription drug
Coupled with the rational/conceptual approach of expert rat- regimen adherence (Axelsson, Brink, Lundgren, &
ing is the empirical approach of applying the nomological Lötvall, 2011; Christensen & Smith, 1995). Thus, machine
network (Cronbach & Meehl, 1955), meaning that perhaps learning algorithms may offer exciting opportunities to im-
we can ‘back into’ personality constructs by demonstrating prove human health and well‐being through personalized in-
sensible big data patterns of convergent and discriminant va- terventions, using big data to detect and influence
lidity. Of course, such work is easier said than done, as all personality, with the goal of forecasting and improving
this presumes that machine learning algorithms and data at real‐world health outcomes (e.g. within health risk evalua-
least lean towards being interpretable. Perhaps that leaning tion, clinical treatment, and public health programmes;
will become even stronger as data scientists understand that Chapman, Hampson, & Clarkin, 2014). In fact, a recent arti-
construct validity remains very important (Bleidorn & Hop- cle demonstrated that adding personality measures to a ma-
wood, 2019; Tay, Woo, Hickman, & Saef, in press)—per- chine learning medical risk model predicting log hazard
haps even more important than ever before given the nature rates of mild cognitive impairment significantly improved
of the data. the model’s overall prediction measured by Nagelkerke’s

© 2020 European Association of Personality Psychology Eur. J. Pers. (2020)

DOI: 10.1002/per
L. Alexander III et al.

pseudo R2 and discrimination as measured by area under the personalized interventions that improve many important
receiver operating characteristic curve (Chapman, Lin, Roy, outcomes, while realizing that more empirical work investi-
Benedict, & Lyness, 2019). gating this hope is desperately needed.
Robust out‐of‐sample prediction of potentially complex
relationships is obviously a major strength of machine learn-
Fairness
ing and big data, as we have noted; yet we should not always
rely on those ‘black box’ predictions alone, because we usu- Fairness issues are critical to consider when capturing
ally want to know why prediction happens and not just that it personality data, applying machine learning, and then
happens. Thus, the trade‐offs between interpretability and reporting and acting on analytic results. Although Internet
prediction must be understood and managed when deciding access penetrates the majority of the US population and
among machine learning and traditional analysis options. has been described as a ‘basic utility for social inclusion’
Fortunately, however, some recent attention in machine in technology‐centric societies (Van Deursen & Van
learning has turned to interpreting these black box models Dijk, 2019, p. 355), the type, speed, and consistency of In-
(Ribeiro et al., 2016a). Although there is debate about the ap- ternet access in the world, or even in specific
propriateness of the interpretation of such models, especially neighbourhoods, can certainly be uneven (Marler, 2018).
for high‐stakes applications (Rudin, 2019), if such model in- In fact, decades of attention have been paid to the ‘digital
terpretation can be achieved, it may be useful when divide’, or the general disparities in access to, usage of,
attempting to apply psychometric evaluations of validity to and proficiency in Internet and communication technologies
these analytical techniques, as knowing what variables are across many key demographics such as age (Van
driving predictions is essential to substantive understanding Volkom, Stapley, & Amaturo, 2014), gender (Ching,
and discovery with big data personality measures. Although Basham, & Jang, 2005), race (Jackson et al., 2008), and
still in their infancy, machine learning interpretation socio‐economic status (Huffman, 2018), as well as within
techniques may hold promise for psychology researchers many critical domains of human activity and endeavour,
who wish to draw sound scientific inferences about such as education (Rowsell, Morrell, & Alvermann, 2017),
personality from big data. One such tool, local interpretable employment (Lindsay, 2005), and healthcare (Mackert,
model‐agnostic explanations (Ribeiro, Singh, & Mabry‐Flynn, Champlin, Donovan, & Pounders, 2016).
Guestrin, 2016b), explains any black box model by approxi- These differences in access to Internet and communication
mating the predictions of the original model with a locally technologies serve as an important reminder for personality
interpretable one (a local surrogate model) trained on researchers to be sensitive to the representativeness of big
perturbations of the original data (permuting or removing data. Representativeness here means that using technology
predictors in the model). Interpretability then can better to understand personality in diverse populations requires
emerge from the local region or regions of interest, a useful sampling in a similarly diverse and representative manner,
approach when personality researchers use more complex moving beyond populations that are westernized, educated,
forms of machine learning. industrialized, rich, and democratic (WEIRD; Henrich,
Although we have mentioned some of the obvious eth- Heine, & Norenzayan, 2010).
ical and privacy concerns surrounding big data personality Consequently, being fair in the era of big data begins with
measurement, what if people at risk of obesity, as identified a sensitivity to critical differences in Internet accessibility
by their personality (Sutin, Ferrucci, Zonderman, & and technical proficiency, as well as other regional, linguis-
Terracciano, 2011), could be detected and assisted by big tic, and race/ethnicity demographics of concern to the
data technologies (e.g. healthy food choice or exercise rec- researcher. These factors, and more, can influence how (or
ommendations)? To the extent that personality is malleable, even whether) personality inferences can be made appropri-
could interactive technologies influence the development or ately from big data. The key fairness question to be investi-
expression of personality traits or facets that contribute to gated here is as follows: can personality be inferred in a
successful weight loss in those seeking to lose weight similar way across subgroups of interest? If personality con-
(Soini, Mustajoki, Eriksson, & Lahti, 2018)? This example structs differ in essential ways across subgroups, then many
and others like it are not just academically interesting; they fairness problems arise, for example, it would be challenging
are important because healthcare outside the hospital in- or even unfair to compare subgroups directly with one an-
volves self‐management, a critical component of many other, and criterion‐related validity would mean different
technologies that inherently will be influenced by personal- things for different subgroups.
ity. By understanding personality, these health‐oriented Traditionally, this issue has been of concern, where
technologies can track human wellness and disease even psychometric judgements of construct similarity involve
more effectively (Erdmier, Hatcher, & Lee, 2016; Jeong, examining factor structure and measurement invariance
Bychkov, & Searson, 2019), and they can be tailored to across subgroups of interest (Chiorri, Marsh, Ubbiali, &
be more responsive and useful. Of course, there are also Donati, 2016; Marsh et al., 2010). This traditional approach
many applications outside of the healthcare industry where could be usefully extended to personality‐relevant big data.
personality‐based interventions might improve people’s In traditional measurement, psychometricians will often first
quality of life at both the individual and societal levels. In- undertake conceptual steps (e.g. reviewing item content for
deed, one can imagine that the detection and/or manipula- construct relevance and cultural sensitivity) and empirical
tion of personality could be usefully exploited to create steps (e.g. investigating measurement invariance and

© 2020 European Association of Personality Psychology Eur. J. Pers. (2020)

DOI: 10.1002/per
 Using Big Data and Machine Learning in Personality Measurement: Opportunities and Challenges

predictive invariance), to determine whether personality be neutral or negative (e.g. when data and algorithmic
measures tend to capture traits reliably and exhibit empirical predictions are being sold to the highest bidder or when
relationships in the same way across subgroups negative information might undermine potential job
(Millsap, 2011). Even though the goals of measurement prospects). Or, being inured to big data privacy issues,
and predictive invariance cannot be simultaneously fulfilled they might not change their behaviour at all under the
in the strictest mathematical sense (Millsap, 2007), it usually assumption that the privacy of their data always runs the
remains practically informative to determine how closely risk of compromise.
these goals are met.
To the extent there is strong evidence against measure-
ment invariance, for instance, then mean differences between MEETING THE CHALLENGES OF BIG DATA AND
subgroups become more driven by the quirks of the items MACHINE LEARNING IN PERSONALITY
than by the construct. And with evidence against predictive RESEARCH
invariance, an overall regression line will make systematic
overprediction and underprediction for the target subgroups Researchers outside of the field of psychology have
of interest that should also be considered. Granted, the experience using a wide variety of data analytic techniques
approach and application of invariance analyses will look for modelling complex systems that might prove valuable
different when based on big data and related algorithms than in measuring personality with big data (see Gerlach
when based on traditional personality measures (e.g. et al., 2018 for an example of a collaboration
applying some form of within‐and‐between group network between engineers, psychologists, and physicists to identify
analysis to personality items instead of a CFA). But the point personality types using big data). Overall, it seems clear that
is that without measurement invariance analyses of personality psychologists are absolutely essential to the work
some sort, construct‐level comparisons between subgroups of data scientists, if they are not data scientists themselves,
on personality‐relevant big data are compromised, as well because big data and machine learning algorithms critically
as a deeper understanding about the substantive nature of al- depend on what traditional research depends on: framing
gorithmic bias (construct irrelevance) in big data. But if we the personality modelling problem appropriately (e.g.
do not even know what or how much variance observed is clustering versus prediction) and selecting construct‐relevant
construct relevant (per the discussion earlier on reliability and reliable measures and outcomes (e.g. personality and
and validity), then we cannot begin to investigate measure- other psychological characteristics; Flake & Fried, 2019).
ment and predictive invariance. Only then can the research and results be not only analyti-
cally sound but also context‐appropriate and informative
towards meeting personality research and practice goals
Privacy
(Lodge, Alhadad, Lewis, & Gašević, 2017).
Last but hardly least, big data pose ethical challenges involv- Fortunately, personality researchers bring valuable exper-
ing those who use the data to infer personality, as well as the tise and training that reflect a unique set of skills that we
people from whom the data are collected. Privacy concerns believe will prove of increasing importance towards inter-
have been longstanding in the health sciences (e.g. Murdoch pretable, practical, ethical, and defensible big data analyses:
& Detsky, 2013) and are emerging in the individual privacy e.g. an understanding of personality constructs, the psycho-
domains (e.g. General Data Protection Regulation in metrics training to develop and evaluate measures of those
Europe; Family Educational Rights and Privacy Act, Fair constructs, and study design and ethics training relevant to
Credit Reporting Act, and state laws in the US), but these the context of social sciences research involving human sub-
concerns are relatively new to personality research (although jects. With its long history of concern for ethical principles
for guidance, see Kosinski et al., 2015). Ethical sensitivity such as privacy, confidentiality, and informed consent, psy-
should certainly accompany all psychological big data chologists in general are particularly well‐suited to develop
research (Mansour, 2016), especially considering that only big data personality assessments and to assist in the develop-
a small subset of characteristics may be necessary to ment of guidelines and laws that help to determine the ethical
re‐identify individuals within an anonymized data set applications of big data. Not only have psychometric and
(Rocher, Hendrickx, & de Montjoye, 2019). ethical concerns been a driving force for many psychological
Specific to personality research, note that the presence researchers who study topics such as ethics in psychological
and type of big data technologies that are used will likely testing, diversity and inclusion, and adverse impact; psychol-
change people’s behavior meaningfully, as the perceptions ogists have also helped to codify the findings of these bodies
and reality of compromised privacy and anonymity with of research into practical guidelines, as they have with
regard to big data continue to grow, and as more diverse the development of the Standards for Educational and
data and complex analytic tools come together more often, Psychological Testing [American Educational Research As-
more quickly, and on a broader scale. As people become sociation (AERA), American Psychological Association
aware that their data at home, at work, at school, and online (APA), & National Council on Measurement in Education
are being combined and mined, they may wish to share (NCME), 2014], the Uniform Guidelines on Employee Selec-
their data when the perceived outcome is positive (e.g. tion Procedures (Equal Employment Opportunity Commis-
better house purchase and better medical advice), and wish sion, 1978), and the Principles for the Validation and Use
to keep their data confidential when the outcome is likely to of Personnel Selection Procedures (Society for Industrial

© 2020 European Association of Personality Psychology Eur. J. Pers. (2020)

DOI: 10.1002/per
L. Alexander III et al.

and Organizational Psychology, 2019). Taken together, per- Technology: Understanding the digital technologies that
sonality researchers are in a unique position to understand, give rise to big data
learn from—and teach others—the important and hard les-
5. Collaborating with multidisciplinary colleagues who have
sons involving the balance or trade‐off between the psycho-
expertise and experience in the implications of using
logical informativeness of big data personality approaches
different digital devices, infrastructures, and software to
and the ethical challenges when using such personality‐
collect, manage, store, or otherwise curate big data
relevant technologies.
collections.
However, it is important for a personality researcher to
6. Researching the reliability of big data collections of fea-
understand that when sharing these traditions while engaging
tures across different technology platforms from which
with big data and machine learning, their discipline often will
the same data might be collected (e.g. laptop versus cell
join many others at the table that have their own expertise,
phone; different operating systems).
goals, and unique priorities (e.g. data science, applied
statistics, healthcare, economics, political science, and
human resources management). Thus, a multidisciplinary Algorithm: Adopting and evaluating algorithmic
approach is both challenging and heartening, because more approaches to personality measurement
substantive conversations and context around big data
and machine learning algorithms can be had, giving 7. Collaborating with multidisciplinary colleagues who can
greater meaning and closer criticism around the prediction‐ inform personality research, who have expertise and
versus‐explanation tradeoffs in machine learning (Yarkoni experience in selecting and justifying the analytical
& Westfall, 2017). For personality researchers, prediction approach chosen to analyse data, for example, deciding
and explanation are part of an iterative process designed to between different prediction models, training and tuning
deepen our understanding of the psychological nature of those models, and accurately interpreting their results.
personality and the corresponding habits of human thought, 8. Emphasizing analytic procedures that avoid overfitting
attitudes, and behaviour. models to personality data (e.g. k‐fold cross‐validation
and bootstrapping), which is an emphasis in machine
learning, but can be applied equally usefully in tradi-
FUTURE DIRECTIONS FOR BIG DATA tional modelling.
PERSONALITY RESEARCH 9. Investigating predictive patterns involving personality
measures: statically, in terms of convergent, discrimi-
As we enter this exciting new age of big data, we encourage nant, and criterion‐related validity; and dynamically,
personality researchers to reflect on 14 important aspects of where mediational and multilevel (cross‐level) relation-
big data personality analyses, to create more synergies (and ships can be tested with longitudinal big data.
fewer competitions) between prediction and explanation in 10. Evaluating the nature of big data collected over time—as
the personality domain. well as results from the clustering and prediction algo-
rithms applied dynamically to those data over time—
such that inferences about both populations and person-
Information: Understanding how to approach big data ality can be drawn from big data, and the scope of the
1. Collaborating with multidisciplinary colleagues in quanti- generalizability and malleability of personality can be
tative disciplines (e.g. computer scientists, data scientists, further broadened, understood, and advanced.
applied statisticians, and other social scientists), who have
expertise and experience in contending with big data
Impact: Fairness and ethics
having multiple formats, along with messiness and
missingness, and applying algorithms for clustering and 11. Striving to ensure that personality measurement is fair
dimensionality reduction. across demographic subgroups (e.g. race/ethnicity, gen-
2. Researching the reliability of big data collections of fea- der, and culture), while realizing that fairness is a con-
tures (e.g. extracted social media themes) across time, cept that encompasses broad issues such as cultural
samples, and settings, including collaborating to develop sensitivity, conflicting definitions such as equity versus
appropriate theory and statistical models that estimate merit, and equal opportunities to provide data.
those reliabilities in big data contexts. 12. Detecting and reducing personality‐related biases and
3. Establishing the construct relevance of big data personal- other irrelevancies detected by algorithms. Bias is a sta-
ity indicators being used via machine learning analyses tistical concept, referring to empirically reliable subgroup
(e.g. interpretation of clusters in k‐means clustering or differences in personality that are due to construct irrele-
neural networks) as it can be compared with traditional re- vancies in the data, the models, or their combination.
liability analysis (e.g. interpretation of factors and factor 13. Ensuring that personality data collection (e.g. data
loadings in a CFA). privacy, anonymity, and security) and data use (e.g.
4. Striving for substantive interpretability of personality analysis and interpretation) are sensitive to and consis-
data; for example, develop theory and conduct empirical tent with updated professional, legal, and ethical
studies aimed at exploring and understanding the content standards (e.g. AERA, APA, NCME, 2014; Equal Em-
validity of big data personality predictors. ployment Opportunity Commission, 1978).

© 2020 European Association of Personality Psychology Eur. J. Pers. (2020)

DOI: 10.1002/per
 Using Big Data and Machine Learning in Personality Measurement: Opportunities and Challenges

14. Transparently reporting the process of personality data Aroganam, G., Manivannan, N., & Harrison, D. (2019). Review on
collection and analysis, disclosing and reflecting on wearable technology sensors used in consumer sport applications.
Sensors, 19, 1–26. https://doi.org/10.3390/s19091983
any key limitations alongside any key benefits. Open sci- Arthur, W., Doverspike, D., Muñoz, G. J., Taylor, J. E., & Carr, A.
ence practices can critically assist in improving transpar- E. (2014). The use of mobile devices in high‐stakes remotely
ency (e.g. preregistration, sharing relevant measures and delivered assessments and testing. International Journal of
protocols, and sharing the variable codebook and code Selection and Assessment, 22, 113–123. https://doi.org/10.1111/
for analyses, if not the data set itself). ijsa.12062
Arthur, W., Glaze, R. M., Villado, A. J., & Taylor, J. E. (2010). The
magnitude and extent of cheating and response distortion effects
on unproctored internet‐based tests of cognitive ability and
CONCLUSION personality. International Journal of Selection and Assessment,
18, 1–16. https://doi.org/10.1111/j.1468-2389.2010.00476.x
Arthur, W., Keiser, N. L., & Doverspike, D. (2018). An
This paper has outlined some of the promises and challenges information‐processing‐based conceptual framework of the ef-
for personality researchers interested in developing and fects of unproctored internet‐based testing devices on scores on
collecting big data from new digital technologies, coupled employment‐related assessments and tests. Human Performance,
with the machine learning algorithms applied to big data 31, 1–32. https://doi.org/10.1080/08959285.2017.1403441
Axelsson, M., Brink, E., Lundgren, J., & Lötvall, J. (2011). The in-
(and most any psychological data). For over a century, psy- fluence of personality traits on reported adherence to medication
chological researchers have accumulated the wisdom from in individuals with chronic disease: An epidemiological study in
hard‐won lessons concerning the measure development, West Sweden. PLoS ONE, 6, 1–7. https://doi.org/10.1371/jour-
analysis, and evaluation process, for example, generating nal.pone.0018241
item content; refining measures psychometrically; examining Azucar, D., Marengo, D., & Settanni, M. (2018). Predicting the Big
5 personality traits from digital footprints on social media: A
correlational patterns with other measures, manipulations, meta‐analysis. Personality and Individual Differences, 124,
and real‐world outcomes. This wisdom seems hardly out- 150–159. https://doi.org/10.1016/j.paid.2017.12.018
dated in the big data era—perhaps it is even more impor- Bachrach, Y., Kosinski, M., Graepel, T., Kohli, P., & Stillwell, D.
tant—and much of it is found in codified professional (2012). Personality and patterns of Facebook usage. Proceed-
standards of measurement, as found in the Standards for Ed- ings of the 4th Annual ACM Web Science Conference, 24–32.
https://doi.org/10.1145/2380718.2380722
ucational and Psychological Testing (AERA, APA, Barocas, S., & Selbst, A. D. (2016). Big data’s disparate impact.
NCME, 2014), now in its third edition. Personality re- SSRN Electronic Journal, 104, 671–732. https://doi.org/
searchers can judge for themselves, as they are engaged in 10.2139/ssrn.2477899
collecting personality‐related big data with new (intensive, Beck, E. D., & Jackson, J. J. (2019a). Within‐person variability.
unobtrusive) data‐collection technologies and sophisticated PsyArXiv. Advance online publication.. https://doi.org/10.31234/
osf.io/kavbp
clustering and predictive algorithms, whether they will likely Beck, E. D., & Jackson, J. J. (2019b). Consistency and change in
benefit from revisiting and making use of our psychometric idiographic personality: A longitudinal ESM network study.
concepts and tools. Journal of Personality and Social Psychology. Advance online
Concluding with an even bigger picture, we would argue publication, 118, 1080–1100. https://doi.org/10.1037/
that big data not only has much to offer to personality psy- pspp0000249
Becker, B. J., & Aloe A. M. (2008). A framework for generalization
chology, but also vice versa, meaning that it is critical for in meta‐analysis: Medical and social science examples [invited
the expertise, accomplishments, and history of personality presentation]. The 16th Merck‐Temple conference on biostatis-
psychology to shape how the big data and analytics commu- tics, Philadelphia, PA.
nity can more effectively and efficiently forge useful path- Beckmann, N., & Wood, R. E. (2017). Editorial: Dynamic personal-
ways towards achieving new and rapid insights about ity science. Integrating between‐person stability and
within‐person change. Frontiers in Psychology, 8, 1–7. https://
personality (habits of human attitudes, thoughts, and behav- doi.org/10.3389/fpsyg.2017.01486
iour). To that end, personality psychologists must continue Bleidorn, W., & Hopwood, C. J. (2019). Using machine learning to
to build and play a part in multidisciplinary communities of advance personality assessment and theory. Personality and So-
interest focused on future developments, applications, and cial Psychology Review, 23, 190–203. https://doi.org/10.1177/
evaluations of big data and machine learning. Trends seem 1088868318772990
Bleidorn, W., Hopwood, C. J., & Lucas, R. E. (2018). Life events
to be moving in this direction—perhaps we could apply ma- and personality trait change. Journal of Personality, 86, 83–96.
chine learning algorithms to big data personality researchers https://doi.org/10.1111/jopy.12286
themselves to accelerate that trend. Bleidorn, W., Hopwood, C. J., & Wright, A. G. (2017). Using big
data to advance personality theory. Current Opinion in Behav-
ioral Sciences, 18, 79–82. https://doi.org/10.1016/j.
cobeha.2017.08.004
REFERENCES Borsboom, D., Mellenbergh, G. J., & Van Heerden, J. (2004). The
concept of validity. Psychological Review, 111, 1061–1071.
Adjerid, I., & Kelley, K. (2018). Big data in psychology: A frame- https://doi.org/10.1037/0033-295X.111.4.1061
work for research advancement. American Psychologist, 73, Campbell, D. T., & Fiske, D. W. (1959). Convergent and discrimi-
899–917. https://doi.org/10.1037/amp0000190 nant validation by the multitrait‐multimethod matrix. Psycholog-
American Educational Research Association, American Psycholog- ical Bulletin, 56, 81–105. https://doi.org/10.1037/h0046016
ical Association, & National Council on Measurement in Educa- Cascio, W., Boudreau, J., & Fink, A. (2019). Investing in people:
tion (2014). Standards for educational and psychological testing. Financial impact of human resource initiatives (3rd ed.). Alexan-
Washington, DC: American Educational Research Association. dria, VA: Society for Human Resource Management.

© 2020 European Association of Personality Psychology Eur. J. Pers. (2020)

DOI: 10.1002/per
L. Alexander III et al.

Cattell, R. B. (1946). Personality structure and measurement. I. The of positive psychology (pp. 35–54). Dordrecht, Netherlands:
operational determination of trait unities. British Journal of Springer.
Psychology, 36, 88–103. https://doi.org/10.1111/j.2044- De Mauro, A., Greco, M., & Grimaldi, M. (2015). What is big data?
8295.1946.tb01110.x A consensual definition and a review of key research topics. AIP
Chapman, B. P., Hampson, S., & Clarkin, J. (2014). Personality‐in- Conference Proceedings, 1644, 97–104. https://doi.org/10.1063/
formed interventions for healthy aging: Conclusions from a Na- 1.4907823
tional Institute on Aging work group. Developmental Domingos, P. (2012). A few useful things to know about machine
Psychology, 50, 1426–1441. https://doi.org/10.1037/a0034135 learning. Communications of the ACM, 55, 78–87. https://doi.
Chapman, B. P., Lin, F., Roy, S., Benedict, R. H. B., & Lyness, J. M. org/10.1145/2347736.2347755
(2019). Health risk prediction models incorporating personality Donnellan, M. B., & Robins, R. W. (2010). Resilient,
data: Motivation, challenges, and illustration. Personality Disor- overcontrolled, and undercontrolled personality types:
ders, Theory, Research, and Treatment, 10, 46–58. https://doi. Issues and controversies. Social and Personality Psychology
org/10.1037/per0000300 Compass, 4, 1070–1083. https://doi.org/10.1111/j.1751-
Chen, E. E., & Wojcik, S. P. (2016). A practical guide to big data 9004.2010.00313.x
research in psychology. Psychological Methods, 21, 458–474. Epskamp, S., Rhemtulla, M., & Borsboom, D. (2017). Generalized
https://doi.org/10.1037/met0000111 network psychometrics: Combining network and latent variable
Chester, D. S., & Lasko, E. N. (in press). Construct validation of ex- models. Psychometrika; Williamsburg, 82, 904–927. https://doi.
perimental manipulations in social psychology: Current practices org.ezproxy.rice.edu/10.1007/s11336-017-9557-x
and recommendations for the future. Perspectives on Psycholog- Epstein, S. (1983). Aggregation and beyond: Some basic issues on
ical Science. the prediction of behavior. Journal of Personality, 51, 360–392.
Cheung, M. W.‐L., & Jak, S. (2016). Analyzing big data in psychol- https://doi.org/10.1111/j.1467-6494.1983.tb00338.x
ogy: A split/analyze/meta‐analyze approach. Frontiers in Psy- Equal Employment Opportunity Commission (1978). Uniform
chology, 7, 1–13. https://doi.org/10.3389/fpsyg.2016.00738 guidelines on employee selection procedures. Federal Register,
Ching, C. C., Basham, J. D., & Jang, E. (2005). The legacy of the 43, 38295–38309.
digital divide: Gender, socioeconomic status, and early exposure Erdmier, C., Hatcher, J., & Lee, M. (2016). Wearable device impli-
as predictors of full‐spectrum technology use among young cations in the healthcare industry. Journal of Medical Engineer-
adults. Urban Education, 40, 394–411. https://doi.org/10.1177/ ing & Technology, 40, 141–148. https://doi.org/10.3109/
0042085905276389 03091902.2016.1153738
Chiorri, C., Marsh, H. W., Ubbiali, A., & Donati, D. (2016). Testing Fan, J., & Li, R. (2006). Statistical challenges with high dimension-
the factor structure and measurement invariance across gender of ality: Feature selection in knowledge discovery. ArXiv:Math. Re-
the Big Five Inventory through exploratory structural equation trieved from. https://arxiv.org/abs/math/0602133
modeling. Journal of Personality Assessment, 98, 88–99. Fiedler, K. (2018). The creative cycle and the growth of psycholog-
https://doi.org/10.1080/00223891.2015.1035381 ical science. Perspectives on Psychological Science, 13,
Chittaranjan, G., Blom, J., & Gatica‐Perez, D. (2013). Mining 433–438. https://doi.org/10.1177/1745691617745651
large‐scale smartphone data for personality studies. Personal Flake, J. K., & Fried, E. I. (2019). Measurement schmeasurement:
and Ubiquitous Computing, 17, 433–450. https://doi.org/ Questionable measurement practices and how to avoid them.
10.1007/s00779-011-0490-1 PsyArXiv. Advance online publication.. https://doi.org/10.31234/
Christensen, A. J., & Smith, T. W. (1995). Personality and patient osf.io/hs7wm
adherence: Correlates of the five‐factor model in renal dialysis. Fleeson, W. (2004). Moving personality beyond the
Journal of Behavioral Medicine, 18, 305–313. https://doi.org/ person‐situation debate: The challenge and the opportunity of
10.1007/BF01857875 within‐person variability. Current Directions in Psychological
Christensen, A. P., Golino, H., & Silvia, P. J. (2020). A psychomet- Science, 13, 83–87. https://doi.org/10.1111/j.0963-7214.2004.
ric network perspective on the validity and validation of person- 00280.x
ality trait questionnaires. European Journal of Personality. Foster, K., Schuh, S., & Zhang, H. (2013). The 2010 survey of con-
Advance online publication. https://doi.org/10.1002/per.2265 sumer payment choice. Research Reviews, 20, 113–118. https://
Corker, K. S., Oswald, F. L., & Donnellan, M. B. (2012). Conscien- doi.org/10.2139/ssrn.2564172
tiousness in the classroom: A process explanation. Journal of Friedman, J., Hastie, T., & Tibshirani, R. (2008). Sparse inverse co-
Personality, 80, 995–1028. https://doi.org/10.1111/j.1467- variance estimation with the graphical lasso. Biostatistics, 9,
6494.2011.00750.x 432–441. https://doi.org/10.1093/biostatistics/kxm045
Costantini, G., Richetin, J., Preti, E., Casini, E., Epskamp, S., & Gandomi, A., & Haider, M. (2015). Beyond the hype: Big data con-
Perugini, M. (2019). Stability and variability of personality net- cepts, methods, and analytics. International Journal of Informa-
works. A tutorial on recent developments in network psychomet- tion Management, 35, 137–144. https://doi.org/10.1016/j.
rics. Personality and Individual Differences, 136, 68–78. https:// ijinfomgt.2014.10.007
doi.org/10.1016/j.paid.2017.06.011 Gerlach, M., Farb, B., Revelle, W., & Amaral, L. A. N. (2018). A
Cronbach, L. J. (1982). Designing evaluations of educational and robust data‐driven approach identifies four personality types
social programs. San Francisco, CA: Jossey‐Bass. https://doi. across four large data sets. Nature Human Behaviour, 2,
org/10.1177/109821408300400210 735–742. https://doi.org/10.1038/s41562-018-0419-z
Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1972). Gladstone, J. J., Matz, S. C., & Lemaire, A. (2019). Can psycholog-
The dependability of behavioral measurements: Theory of gener- ical traits be inferred from spending? Evidence from transaction
alizability for scores and profiles. New York, NY: John Wiley & data. Psychological Science, 30, 1087–1096. https://doi.org/
Sons. https://doi.org/10.1126/science.178.4067.1275 10.1177/0956797619849435
Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psy- Golbeck, J., Robles, C., Edmondson, M., & Turner, K. (2011).
chological tests. Psychological Bulletin, 52, 281–302. https:// Predicting personality from Twitter. 2011 IEEE Third Interna-
doi.org/10.1037/h0040957 tional Conference on Privacy, Security, Risk and Trust and
Cronbach, L. J., Rajaratnam, N., & Gleser, G. C. (1963). Theory of 2011 IEEE Third International Conference on Social Computing,
generalizability: A liberalization of reliability theory. British 149–156. https://doi.org/10.1109/PASSAT/SocialCom.2011.33
Journal of Statistical Psychology, 16, 137–163. https://doi.org/ Golbeck, J., Robles, C., & Turner, K. (2011). Predicting personality
10.1111/j.2044-8317.1963.tb00206.x with social media. CHI EA’11: CHI’11 Extended Abstracts on
Csikszentmihalyi, M., & Larson, R. (2014). Validity and reliability Human Factors in Computing Systems, 253–262. https://doi.
of the experience‐sampling method. In Flow and the foundations org/10.1145/1979742.1979614

© 2020 European Association of Personality Psychology Eur. J. Pers. (2020)

DOI: 10.1002/per
 Using Big Data and Machine Learning in Personality Measurement: Opportunities and Challenges

Hamaker, E. L., Nesselroade, J. R., & Molenaar, P. C. (2007). The Kuhn, M. (2019). caret: Classification and regression training. R
integrated trait‐state model. Journal of Research in Personality, package version 6.0–84. https://CRAN.R-project.org/
41, 295–315. https://doi.org/10.1016/j.jrp.2006.04.003 package¼caret
Haqiqatkhah, M. M., & Tuerlinckx, F. (2019). Are we on the same Landers, R. N., Brusso, R. C., Cavanaugh, K. J., & Collmus, A. B.
page? Latent variable modeling suggests different nomothetic (2016). A primer on theory‐driven web scraping: Automatic ex-
and idiographic factor structures for momentary affect. PsyArXiv. traction of big data from the internet for use in psychological re-
Advance online publication. https://doi.org/10.31234/osf.io/ search. Psychological Methods, 21, 475–492. https://doi.org/
6wsgd 10.1037/met0000081
Harari, G. M., Lane, N. D., Wang, R., Crosier, B. S., Campbell, A. T., Laney, D. (2001, February 6). 3‐D Data management: Controlling
& Gosling, S. D. (2016). Using smartphones to collect behavioral data volume, velocity, and variety. Application Delivery Strate-
data in psychological science: Opportunities, practical consider- gies by META Group Inc. Retrieved from https://blogs.gartner.
ations, and challenges. Perspectives on Psychological Science, com/doug-laney/files/2012/01/ad949-3D-Data-Management-
11, 838–854. https://doi.org/10.1177/1745691616650285 Controlling-Data-Volume-Velocity-and-Variety.pdf
Hawkins, D. M. (2004). The problem of overfitting. Journal of Lenhart, A., Duggan, M., Perrin, A., Steppler, R., Rainie, L., & Par-
Chemical Information and Computer Sciences, 44, 1–12. ker, K. (2015). Teens, social media, & technology overview 2015:
https://doi.org/10.1021/ci0342472 Smartphones facilitate shifts in communication landscape for
Henrich, J., Heine, S. J., & Norenzayan, A. (2010). The weirdest teens (p. 48). Retrieved from https://www.pewresearch.org/wp-
people in the world? Behavioral and Brain Sciences, 33, content/uploads/sites/9/2015/04/PI_TeensandTech_Update2015_
61–83. https://doi.org/10.1017/S0140525X0999152X 0409151.pdf
Huffman, S. (2018). The digital divide revisited: What is next? Ed- Lindsay, C. (2005). Employability, services for unemployed job
ucation, 138, 239–246. Retrieved from. https://search-ebscohost- seekers and the digital divide. Urban Studies, 42, 325–339.
com.ezproxy.rice.edu https://doi.org/10.1080/0042098042000316173
Iacobelli, F., Gill, A. J., Nowson, S., & Oberlander, J. (2011). Large Lodge, J. M., Alhadad, S. S. J., Lewis, M. J., & Gašević, D. (2017).
scale personality classification of bloggers. In S. D’Mello, A. Inferring learning from big data: The importance of a transdisci-
Graesser, B. Schuller, & J.‐C. Martin (Eds.), Affective computing plinary and multidimensional approach. Technology, Knowledge
and intelligent interaction (pp. 568–577). https://doi.org/ and Learning, 22, 385–400. https://doi.org/10.1007/s10758-
10.1007/978-3-642-24571-8_71 017-9330-3
Ihsan, Z., & Furnham, A. (2018). The new technologies in person- Lüdtke, O., Roberts, B. W., Trautwein, U., & Nagy, G. (2011). A
ality assessment: A review. Consulting Psychology Journal: random walk down university avenue: Life paths, life events,
Practice and Research, 70, 147–166. https://doi.org/10.1037/ and personality trait change at the transition to university life.
cpb0000106 Journal of Personality and Social Psychology, 101, 620–637.
Ilies, R., & Judge, T. A. (2002). Understanding the dynamic rela- https://doi.org/10.1037/a0023743
tionships among personality, mood, and job satisfaction: A field Mackert, M., Mabry‐Flynn, A., Champlin, S., Donovan, E. E., &
experience sampling study. Organizational Behavior and Human Pounders, K. (2016). Health literacy and health information tech-
Decision Processes, 89, 1119–1139. https://doi.org/10.1016/ nology adoption: The potential for a new digital divide. Journal
S0749-5978(02)00018-3 of Medical Internet Research, 18, 211–226. https://doi.org/
Inoubli, W., Aridhi, S., Mezni, H., Maddouri, M., & Nguifo, E. M. 10.2196/jmir.6349
(2018). An experimental survey on big data frameworks. Future Mansour, R. F. (2016). Understanding how big data leads to social
Generation Computer Systems, 86, 546–564. https://doi.org/ networking vulnerability. Computers in Human Behavior, 57,
10.1016/j.future.2018.04.032 348–351. https://doi.org/10.1016/j.chb.2015.12.055
Ippel, L., Kaptein, M. C., & Vermunt, J. K. (2019). Estimating mul- Mardonova, M., & Choi, Y. (2018). Review of wearable device
tilevel models on data streams. Psychometrika, 84, 41–64. technology and its applications to the mining industry. Energies,
https://doi.org/10.1007/s11336-018-09656-z 11, 1–14. https://doi.org/10.3390/en11030547
Jackson, L. A., Zhao, Y., Kolenic, A. III, Fitzgerald, H. E., Harold, Marler, W. (2018). Mobile phones and inequality: Findings, trends,
R., & Von Eye, A. (2008). Race, gender, and information technol- and future directions. New Media & Society, 20, 3498–3520.
ogy use: The new digital divide. Cyberpsychology & Behavior, https://doi.org/10.1177/1461444818765154
11, 437–442. https://doi.org/10.1089/cpb.2007.0157 Marsh, H. W., Lüdtke, O., Muthén, B., Asparouhov, T., Morin, A. J.
James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An intro- S., Trautwein, U., & Nagengast, B. (2010). A new look at the big
duction to statistical learning. New York, NY: Springer. https:// five factor structure through exploratory structural equation
doi.org/10.1007/978-1-4614-7138-7 modeling. Psychological Assessment, 22, 471–491. https://doi.
Jeong, I. C., Bychkov, D., & Searson, P. C. (2019). Wearable de- org/10.1037/a0019227
vices for precision medicine and health state monitoring. IEEE McCrae, R. R., & John, O. P. (1992). An introduction to the
Transactions on Biomedical Engineering, 66, 1242–1258. five‐factor model and its applications. Journal of Personality, 60,
https://doi.org/10.1109/TBME.2018.2871638 175–215. https://doi.org/10.1111/j.1467-6494.1992.tb00970.x
Judge, T. A., Simon, L. S., Hurst, C., & Kelley, K. (2014). What I Meade, A. W., Michels, L. C., & Lautenschlager, G. J. (2007). Are
experienced yesterday is who I am today: Relationship of work internet and paper‐and‐pencil personality tests truly comparable?:
motivations and behaviors to within‐individual variation in the An experimental design measurement invariance study. Organi-
five‐factor model of personality. Journal of Applied Psychology, zational Research Methods, 10, 322–345. https://doi.org/
99, 199–221. https://doi.org/10.1037/a0034485 10.1177/1094428106289393
Kelley, T. L. (1927). Interpretation of educational measurements. Meers, K., Dejonckheere, E., Kalokerinos, E. K., Rummens, K., &
New York, NY: World Book Company. Kuppens, P. (2019, June 12). mobileQ: A free user‐friendly appli-
Kosinski, M., Matz, S. C., Gosling, S. D., Popov, V., & Stillwell, D. cation for collecting experience sampling data. https://doi.org/
(2015). Facebook as a research tool for the social sciences: Op- 10.31234/osf.io/ynj7e
portunities, challenges, ethical considerations, and practical Millsap, R. E. (2007). Invariance in measurement and prediction
guidelines. American Psychologist, 70, 543–556. https://doi. revisited. Psychometrika, 72, 461–473. https://doi.org/10.1007/
org/10.1037/a0039210 s11336-007-9039-7
Kosinski, M., Wang, Y., Lakkaraju, H., & Leskovec, J. (2016). Min- Millsap, R. E. (2011). Statistical approaches to measurement in-
ing big data to extract patterns and predict real‐life outcomes. variance. New York, NY: Taylor & Francis.
Psychological Methods, 21, 493–506. https://doi.org/10.1037/ Minbashian, A., Wood, R. E., & Beckmann, N. (2010). Task‐contin-
met0000105 gent conscientiousness as a unit of personality at work. Journal of

© 2020 European Association of Personality Psychology Eur. J. Pers. (2020)

DOI: 10.1002/per
L. Alexander III et al.

Applied Psychology, 95, 793–806. https://doi.org/10.1037/ (2013). Personality, gender, and age in the language of social me-
a0020016 dia: The open‐vocabulary approach. PLoS ONE, 8, 1–16. https://
Molenaar, P. C., & Campbell, C. G. (2009). The new doi.org/10.1371/journal.pone.0073791
person‐specific paradigm in psychology. Current Directions in Settanni, M., Azucar, D., & Marengo, D. (2018). Predicting
Psychological Science, 18, 112–117. https://doi.org/10.1111/ individual characteristics from digital traces on social
j.1467-8721.2009.01619.x media: A meta‐analysis. CyberPsychology, Behavior & Social
Mønsted, B., Mollgaard, A., & Mathiesen, J. (2018). Phone‐based Networking, 21, 217–228. https://doi.org/10.1089/
metric as a predictor for basic personality traits. Journal of Re- cyber.2017.0384
search in Personality, 74, 16–22. https://doi.org/10.1016/j. Simons, D. J., Shoda, Y., & Lindsay, D. S. (2017). Constraints on
jrp.2017.12.004 generality (COG): A proposed addition to all empirical papers.
Murdoch, T. B., & Detsky, A. S. (2013). The inevitable application Perspectives on Psychological Science, 12, 1123–1128. https://
of big data to health care. JAMA, 309, 1351–1352. https://doi. doi.org/10.1177/1745691617708630
org/10.1001/jama.2013.393 Society for Industrial and Organizational Psychology (2019). Prin-
Nosek, B. A., & Errington, T. M. (2020). What is replication? PLoS ciples for the validation and use of personnel selection proce-
Biology, 18, 1–8. https://doi.org/10.1371/journal.pbio.3000691 dures. Industrial and Organizational Psychology, 11, 1–97.
Ones, D. S., & Wiernik, B. M. (2018, October 10). On “new” per- https://doi.org/10.1017/iop.2018.195
sonality types. Retrieved from https://www.siop.org/Research- Soini, S., Mustajoki, P., Eriksson, J. G., & Lahti, J. (2018). Person-
Publications/Items-of-Interest/ArtMID/19366/ArticleID/1698/ ality traits associated with weight maintenance among successful
On-%E2%80%9CNew%E2%80%9D-Personality-Types weight losers. American Journal of Health Behavior, 42, 78–84.
Perrin, A., & Anderson, M. (2019, April 10). Share of U.S. adults https://doi.org/10.5993/AJHB.42.6.8
using social media, including Facebook, is mostly unchanged Stillwell, D. J., & Kosinski, M. (2020, May 1). myPersonality pro-
since 2018. Retrieved May 27, 2019, from Pew Research Center ject website. Retrieved from http://mypersonality.org/
website: https://www.pewresearch.org/fact-tank/2019/04/10/ Storey, V. C., & Song, I.‐Y. (2017). Big data technologies and man-
share-of-u-s-adults-using-social-media-including-facebook-is- agement: What conceptual modeling can do. Data & Knowledge
mostly-unchanged-since-2018/ Engineering, 108, 50–67. https://doi.org/10.1016/j.
Quercia, D., Kosinski, M., Stillwell, D., & Crowcroft, J. (2011). datak.2017.01.001
Our Twitter profiles, our selves: Predicting personality with Twit- Strickhouser, J., Zell, E., & Krizan, Z. (2017). Does personality pre-
ter. 2011 IEEE Third International Conference on Privacy, Secu- dict health and well‐being? A metasynthesis. Health Psychology,
rity, Risk and Trust and 2011 IEEE Third International 36, 797–810. https://doi.org/10.1037/hea0000475
Conference on Social Computing, 180–185. https://doi.org/ Sumner, C., Byers, A., Boochever, R., & Park, G. J. (2012).
10.1109/PASSAT/SocialCom.2011.26 Predicting dark triad personality traits from Twitter usage and
Rentfrow, P. J., & Gosling, S. D. (2003). The do re mi’s of everyday a linguistic analysis of tweets. 2012 11th International Confer-
life: The structure and personality correlates of music prefer- ence on Machine Learning and Applications, 2, 386–393.
ences. Journal of Personality and Social Psychology, 84, https://doi.org/10.1109/ICMLA.2012.218
1236–1254. https://doi.org/10.1037/0022-3514.84.6.1236 Sumner, C., Byers, A., & Shearing, M. (2011). Determining person-
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016a). Model‐agnostic ality traits & privacy concerns from Facebook activity. Black Hat
interpretability of machine learning. In B. Kim, D. M. Malioutov, Briefings, 11, 197–221.
& K. R. Varshney (Eds.), Proceedings of the 2016 ICML Work- Sutin, A. R., Ferrucci, L., Zonderman, A. B., & Terracciano, A.
shop on Human Interpretability in Machine Learning (2011). Personality and obesity across the adult life span. Journal
(pp. 91–95). https://doi.org/10.1145/2939672.2939778 of Personality and Social Psychology, 101, 579–592. https://doi.
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016b). “Why should I org/10.1037/a0024286
trust you?”: Explaining the predictions of any classifier. Proceed- Tay, L., Woo, S. E., Hickman, L., & Saef, R. (in press). Psychomet-
ings of the 22nd ACM SIGKDD International Conference on ric and validity issues in machine learning approaches to person-
Knowledge Discovery and Data Mining, 1135–1144. https:// ality assessment: A focus on social media text mining. European
doi.org/10.1145/2939672.2939778 Journal of Personality. https://doi.org/10.1002/per.2290
Roberts, B. W., Kuncel, N. R., Shiner, R., Caspi, A., & Goldberg, L. Thorndike, E. L. (1904). An introduction to the theory of mental and
R. (2007). The power of personality: The comparative validity of social measurements. New York, NY: Columbia University
personality traits, socioeconomic status, and cognitive ability for Press.
predicting important life outcomes. Perspectives on Psychologi- Tibshirani, R. (1996). Regression shrinkage and selection via the
cal Science, 2, 313–345. https://doi.org/10.1111/j.1745- lasso. Journal of the Royal Statistical Society. Series B Methodo-
6916.2007.00047.x logical, 58, 267–288. https://doi.org/10.1111/j.2517-6161.1996.
Rocher, L., Hendrickx, J. M., & de Montjoye, Y.‐A. (2019). Esti- tb02080.x
mating the success of re‐identifications in incomplete datasets Tskhay, K. O., & Rule, N. O. (2014). Perceptions of personality in
using generative models. Nature Communications, 10, 1–9. text‐based media and OSN: A meta‐analysis. Journal of Research
https://doi.org/10.1038/s41467-019-10933-3 in Personality, 49, 25–30. https://doi.org/10.1016/j.
Roethlisberger, F. J., & Dickson, W. J. (1939). Management and the jrp.2013.12.004
worker. Oxford, England: Harvard University Press. Umaki, T. M., Umaki, M. R., & Cobb, C. M. (2012). The psychol-
Rowsell, J., Morrell, E., & Alvermann, D. E. (2017). Confronting ogy of patient compliance: A focused review of the literature.
the digital divide: Debunking brave new world discourses. Read- Journal of Periodontology, 83, 395–400. https://doi.org/
ing Teacher, 71, 157–165. https://doi.org/10.1002/trtr.1603 10.1902/jop.2011.110344
Rudin, C. (2019). Stop explaining black box machine learning Van Deursen, A. J., & Van Dijk, J. A. (2019). The first‐level digital
models for high stakes decisions and use interpretable models in- divide shifts from inequalities in physical access to inequalities in
stead. Nature Machine Intelligence, 1, 206–215. Retrieved from. material access. New Media & Society, 21, 354–375. https://doi.
https://arxiv.org/abs/1811.10154v3 org/10.1177/1461444818797082
Salgado, J. F., & Moscoso, S. (2003). Internet‐based personality Van Volkom, M., Stapley, J. C., & Amaturo, V. (2014). Revisiting
testing: Equivalence of measures and assesses’ perceptions and the digital divide: Generational differences in technology use in
reactions. International Journal of Selection and Assessment, everyday life. North American Journal of Psychology, 16,
11, 194–205. https://doi.org/10.1111/1468-2389.00243 557–574.
Schwartz, H. A., Eichstaedt, J. C., Kern, M. L., Dziurzynski, L., Wald, R., Khoshgoftaar, T., & Sumner, C. (2012). Machine predic-
Ramones, S. M., Agrawal, M., Shah, A., … Ungar, L. H. tion of personality from Facebook profiles. 2012 IEEE 13th

© 2020 European Association of Personality Psychology Eur. J. Pers. (2020)

DOI: 10.1002/per
 Using Big Data and Machine Learning in Personality Measurement: Opportunities and Challenges

International Conference on Information Reuse Integration, Yost, A. B., Behrend, T. S., Howardson, G., Darrow, J. B., &
109–115. https://doi.org/10.1109/IRI.2012.6302998 Jensen, J. M. (2019). Reactance to electronic surveillance:
Woo, S. E., Tay, L., Jebb, A. T., Ford, M. T., & Kern, M. L. (2020). A test of antecedents and outcomes. Journal of Business and
Big data for enhancing measurement quality. In S. E. Woo, L. Psychology, 34, 71–86. https://doi.org/10.1007/s10869-018-
Tay, & R. W. Proctor (Eds.), Big Data in Psychological Research 9532-2
(pp. 59–85). American Psychological Association. https://doi. Youyou, W., Kosinski, M., & Stillwell, D. (2015). Computer‐based
org/10.1037/0000193-004 personality judgments are more accurate than those made by
Wright, A. G. C. (2014). Current directions in personality science humans. Proceedings of the National Academy of Sciences, 112,
and the potential for advances through computing. IEEE Transac- 1036–1040. https://doi.org/10.1073/pnas.1418680112
tions on Affective Computing, 5, 292–296. https://doi.org/ Youyou, W., Stillwell, D., Schwartz, H. A., & Kosinski, M. (2017).
10.1109/TAFFC.2014.2332331 Birds of a feather do flock together: Behavior‐based personality‐
Yan, Y., Nie, J., Huang, L., Li, Z., Cao, Q., & Wei, Z. (2015). Is assessment method reveals personality similarity among couples
your first impression reliable? Trustworthy analysis using facial and friends. Psychological Science, 28, 276–284. https://doi.
traits in portraits. In X. He, S. Luo, D. Tao, C. Xu, J. Yang, & org/10.1177/0956797616678187
M. A. Hasan (Eds.), Multimedia modeling (pp. 148–158). Zimmermann, J., Woods, W. C., Ritter, S., Happel, M., Masuhr, O.,
https://doi.org/10.1007/978-3-319-14442-9_13 Jaeger, U., … Wright, A. G. C. (2019). Integrating structure and
Yarkoni, T., & Westfall, J. (2017). Choosing prediction over expla- dynamics in personality assessment: First steps toward the devel-
nation in psychology: Lessons from machine learning. Perspec- opment and validation of a personality dynamics diary. Psycho-
tives on Psychological Science, 12, 1100–1122. https://doi.org/ logical Assessment, 31, 516–531. https://doi.org/10.1037/
10.1177/1745691617693393 pas0000625

© 2020 European Association of Personality Psychology Eur. J. Pers. (2020)

DOI: 10.1002/per