Regular Manuscript

The Counseling Psychologist

2022, Vol. 50(7) 918–941
Development and Initial © The Author(s) 2022
Article reuse guidelines:
Validation of the sagepub.com/journals-permissions
DOI: 10.1177/00110000221110847
journals.sagepub.com/home/tcp
Multidimensional Sense
of Emptiness Scale

Hulya Ermis-Demirtas1 , Robert L. Smith2, and

Joshua C. Watson2

Abstract
The present study describes the development and initial validation of the
Multidimensional Sense of Emptiness Scale, a measure based on a theoretically and
empirically grounded conceptualization of emptiness. In the first sample (n = 541),
an exploratory factor analysis yielded three factors, Sense of Inner Emptiness,
Sense of Absence of Relatedness, and Sense of Meaninglessness, explaining 82.8%
of the variance with 13 items. In an additional sample (n = 212), a confirmatory
factor analysis supported this three-factorial solution’s stability. Furthermore, all
subscales were significantly related to a single, second-order factor. In the total
sample (N = 753), subscale and full-scale items offered evidence of satisfactory
internal consistency and convergent validity. We discuss study limitations and
implications for counseling practice, advocacy, education, training, and research.

Keywords
sense of emptiness, factor analysis, reliability, validity, instrument
development

1
Department of Counselor Education, Northeastern Illinois University
2
Department of Counseling & Educational Psychology, Texas A&M University-Corpus Christi

Corresponding Author:
Correspondence concerning this article should be addressed to Hulya Ermis-Demirtas,
Department of Counselor Education at Northeastern Illinois University, 5500 North St. Louis
Ave, Chicago, IL 60625-4699.
Email: [email protected]

The Division 17 logo denotes that this article is designated as a CE article. To purchase the CE
Test, please visit www.apa.org/ed/ce.
Ermis-Demirtas et al. 919

Significance of the Scholarship to the Public

This research explores the state of emptiness to understand the feelings
and core markers associated with this experience and develop a self-
report instrument to measure this construct. In this study, we define
emptiness as a sense of disconnection from the emotional, existential,
and relational spheres of experience. This article introduces a new
survey-based measure of emptiness entitled the Multidimensional Sense
of Emptiness Scale, including three components: Sense of Inner Emp-
tiness, Sense of Meaninglessness, and Sense of Absence of Relatedness.

In recent years, the construct of emptiness has received renewed attention

and legitimacy, perhaps due to its relationship with a wide range of mental
health concerns (Elsner et al., 2018; Miller et al., 2020; Peteet, 2011). The
Diagnostic and Statistical Manual of Mental Disorders Fifth Edition
(American Psychiatric Association [APA], 2013) and the Tenth Revision of
the Mental and Behavioral Disorders chapter of the International Statistical
Classification of Diseases and Related Health Problems (World Health
Organization [WHO], 2016) include emptiness as a diagnostic criterion for
several mental health problems, including major depressive disorder (e.g.,
feeling empty, feelings of emptiness) and borderline personality disorder
(chronic feelings of emptiness; APA, 2013). Researchers have also demon-
strated the detrimental impact of emptiness, connecting it to various mental
health conditions, including bulimia nervosa (Meehan, 2007), persistent
complex bereavement disorder (Parkes, 2001), alcohol, drug, and sex ad-
dictions (Roos et al., 2015), dissociation, nonsuicidal self-injury (Rallis et al.,
2012), and adverse childhood experiences (Buggs, 1996). However, empti-
ness is mostly related to suicide (Segal-Engelchin et al., 2015) and considered
one of the most substantial factors in suicidal ideation and behaviors
(Blascallo-Fontecilla et al., 2013; Chia et al., 2008; Klonsky, 2008).
Despite the relationship of emptiness with various mental health concerns,
counseling practitioners still need a clear and consistent understanding and
assessment of this phenomenon (Klonsky, 2008; Peteet, 2011). Conceptu-
alization of this concept warrants additional delineation because the nature of
the term emptiness is blurred (Elsner et al., 2018) and resembles similar
emotional states, such as boredom, numbness, loneliness, and hopelessness.
However, emptiness does not overlap with these constructs (Hazell, 2003;
Klonsky, 2008) due to “a profound lack of emotional depth or sense of not
being in the experience” (Miller et al., 2020, p. 2). Blasco-Fontecilla et al.
(2013) indicated the absence of an up-to-date, standardized, and compre-
hensive measure and a theoretically grounded definition of emptiness on
920 The Counseling Psychologist 50(7)

which such a measure could be based. Therefore, they called attention to

developing a psychometrically sound method to assess the levels of emptiness
efficaciously. Although there are two existing instruments designed to
measure emptiness, they have deficiencies based on their theoretical foun-
dations, the clarity and readability of their items, the samples in which they
were developed, and evidence supporting their validity. Consequently, a valid
and reliable instrument for assessing emptiness is lacking (Blasco-Fontecilla
et al., 2013; Elsner et al., 2018).

Current Assessment Tools and Their Limitations

The nature of emptiness may be elusive and difficult to express in technical
language. Therefore, researchers have attempted to capture the aspects and
identify discrete factors of this concept. Hazell (1984) made the initial attempt
by developing the Experienced Level of Emptiness Scale with a sample of 270
college students. The 17 scale items, which explained 60.3% of the variance,
were generated based on unspecified existing theories of emptiness and in-
terviews with nonclinical adults. Hazell reported a high Cronbach’s alpha (α =
.91) and a strong relationship with depression in a subsample of 24 partic-
ipants (r = .69).
Buggs (1996) made another attempt by developing the Emptiness Scale in
a clinical sample of 201 participants, with 89.5% self-identified as White. This
scale contains 28 items accounting for 60.2% of the variance. Buggs proposed
a two-dimensional conceptualization of emptiness: (a) an inner sense of
emotional hunger and yearning and (b) a generalized sense of emotional
numbness. However, one major factor emerged from the principal component
analysis (PCA), composed of 13 numbness items and nine emotional hunger
and yearning items. The PCA failed to indicate two discrete and stable factors
as initially proposed but instead supported a single major factor that accounted
for 49% of the variance. Unexpectedly, explaining 11.2% of the variance, a
smaller second factor with five items emerged from the analysis, which
mentioned the connection to people, suggesting an aspect of emptiness in-
volving the ability to relate to others.
Hazell (1984) and Buggs (1996) established an empirical foundation with
their pioneering research exploring emptiness. However, the scales they
developed are not without criticism. First, the conceptual clarity of emptiness
in their studies is limited, since the researchers failed to present an explicit
theoretical foundation and precise definition of this construct. Besides, the
instruments included items related to depersonalization, numbness, boredom,
and loneliness, suggesting that they are somewhat broad and lacking clear
distinction from associated constructs. Although Hazell and Buggs identified
different aspects of emptiness, such as an inner sense of hollowness, part of
self-being missing, numbness, and hunger and yearning, they did not
Ermis-Demirtas et al. 921

investigate other possible components, such as the lack of meaning and

purpose or absence of relatedness to others (Miller et al., 2020; Peteet, 2011).
Second, the clarity and readability of the instrument items constitute another
limitation. For example, in the data collection process, Buggs asked partic-
ipants to provide verbal and written comments after completing the ques-
tionnaire. The feedback suggested that the items were not easily understood.
Third, the samples appear to be a source of limitation in these instrument
development studies. They recruited less than 300 respondents; therefore, the
sample size might be questionable for exploratory factor analysis (EFA;
Tabachnick & Fidell, 2019). Additionally, the samples are limited due to the
lack of diversity. In Buggs’s study, White participants constituted 90% of the
sample, raising concerns regarding the generalizability of the results. In
Hazell’s study, the participants’ race was not reported.
Lastly, Hazell (1984) and Buggs (1996) failed to provide sufficient validity
evidence regarding the scales’ internal structure, since they conducted EFA to
explore the scale factorial solution yet did not perform a confirmatory factor
analysis (CFA) to verify the number of underlying latent variables and the
pattern of observed variable-factor relationship (Brown, 2015). Moreover, to
provide further validity evidence, Hazell examined the correlation of the
Experienced Levels of Emptiness Scale (Hazell, 1984) with the Costello-
Comrey Depression Scale (Costello & Comrey, 1967) in a small subsample of
the study (n = 24), and Buggs investigated the relationship of emptiness with
childhood abuse via a self-generated questionnaire rather than using a vali-
dated instrument, thereby limiting the validity evidence regarding relations to
other variables. In sum, prevailing emptiness measures are limited in their
comprehensive assessment of this experience, item clarity, samples, and
construct validity evidence.

Conceptual Framework
Considering the existing literature, presenting a fundamental definition
representing emptiness in a universal sense seems challenging because
scholars use this phenomenon to describe a subjective experience unique to
individuals based on underlying psychodynamics. In addition to a compre-
hensive review of theories, conceptual studies, and empirical research on
emptiness, we investigated the overlapping aspects of this construct that
supported an inclusive multidimensional theoretical model. In this framework,
emptiness is conceptualized as a sense of detachment from the intrapersonal,
existential, interpersonal, and spiritual domains of experience as inferred from
the extant literature (Blasco-Fontecilla et al., 2013; Buggs, 1996; Peteet,
2011), including four distinct perspectives: (a) a sense of inner emptiness, (b) a
sense of meaninglessness, (c) a sense of absence of relatedness, and (d) a sense
of spiritual emptiness.
922 The Counseling Psychologist 50(7)

Frankl (1984) posited that emptiness consists of two states: a sense of inner
emptiness and meaninglessness. In this study, the term sense of inner emp-
tiness represents the emotional aspect of emptiness and refers to a sense of
internal hollowness, difficulty identifying inner feelings, being out of touch
with the inner self, inner void, a part being missing, and emotional hunger and
yearning for something to fill the void. The term sense of meaninglessness, the
existential perspective of emptiness, refers to a sense of lacking meaning,
purpose, and a clear direction in life and perceiving life as meaningless.
Cushman (1990) defined emptiness as lacking community, a tradition, and
shared meaning from the social perspective. Frankl (1969) also supported the
interpersonal aspect of emptiness by describing this state as a sense of longing
to be connected to others. Considering these approaches to emptiness, the
authors have coined the term sense of absence of relatedness. The sense of
absence of relatedness represents a lack of shared meaning in community and
capacity for emotional closeness in social relationships. Lastly, the sense of
spiritual emptiness refers to the detachment from nature, the universe, and a
higher power. There is also a sense of abandonment by the higher power and
the inability to find a deep inner sense of harmony and strength in spirituality
(Arndt et al., 2013).
In a phenomenological study, Dunn (1994) explored the psychological
emptiness in religious women’s lives and revealed three themes along with
subthemes. These themes include (a) breakdown of the traditional structure of
spiritual life, representing the loss of meaning and uncertainty about new
forms of spiritual life, referring to both existential and spiritual aspects of
emptiness; (b) a struggle with intimacy, referring to an inability to develop
intimate relationships and a sense of disconnection from others; and (c) a core-
self psychological emptiness, referring to the fragmentation of the self and a
sense of having no feelings inside. This qualitative inquiry supports the four-
dimensional structure of emptiness, which constituted the conceptual
framework for the present scale development study.

Links of Emptiness to Suicide Risk, Hope, and Presence

of Life Meaning
Emptiness, conceptualized as a sense of detachment from the lived experi-
ence, has been associated with various psychological constructs, yet mostly
with suicide risk (Blasko-Fontecilla et al., 2013; Klonsky, 2008; Segal-
Engelchin et al., 2015). Suicide remains among the most tragic conse-
quences of psychological pain, accounting for 1.6% of all deaths worldwide
and making it the second among persons aged 15 to 24 years old, an age group
typically inclusive of high school through college students (Drapeau &
McIntosh, 2020). Researchers have traditionally focused on identifying po-
tential risk factors for suicide. In multiple qualitative studies, emptiness
Ermis-Demirtas et al. 923

appeared to be an essential element of individuals who attempt or die by

suicide (Chia et al., 2008; Segal-Engelchin et al., 2015). For example, a study
analyzing 398 Singaporean suicide letters’ content illustrated emptiness as the
second most frequent emotional factor expressed for suicide (25%), following
despondency/agony (60%; Chi et al., 2008). These results together indicate
emptiness as a factor contributing to the risk of suicide.
Furthermore, the empirical inquiry regarding the relationship of emptiness
with positive psychology constructs, including hope and meaning in life, has
received less interest in the field, even though researchers have addressed their
association conceptually. Cheng et al. (2013) characterized emptiness as a loss
of hope, meaning, and purpose in life, suggesting the presence of hope and
meaning in life as buffers against this experience. Frankl (1984) has received
credit for the emergence of meaning as an essential psychological construct
(Steger et al., 2006). Frankl delineated a search for meaning as the primary
motivation and stated that the failure in finding a significant meaning in life
could result in meaninglessness and inner emptiness.

The Present Study

The purpose of this research was to address a significant gap in the existing
instruments assessing emptiness by creating a reliable and valid multidi-
mensional measure encompassing aspects of emptiness, including a sense of
inner emptiness, meaninglessness, absence of relatedness, and spiritual
emptiness. To accomplish this goal, using a more practitioner-friendly vo-
cabulary, we engaged in the initial development and validation of a measure,
named Multidimensional Sense of Emptiness Scale (MSES) in the general
population, which would allow the use of this instrument for research, as-
sessment, and screening purposes. The present study also examined the re-
lationship of the MSES with the Suicide Probability Scale (SPS; Cull & Gill,
2002), the Adult Hope Scale (AHS; Snyder et al., 1991), and the Meaning in
Life Questionnaire–Presence Subscale (MLQ-PS; Steger et al., 2006) to
establish convergent validity evidence by determining the direction and
degree of the relationships among these instruments. As such, in this study, we
were guided by the following hypotheses:

Hypothesis 1: We anticipate that the MSES will have a multidimensional

structure with items that describe the following four components: a sense of
inner emptiness, meaninglessness, absence of relatedness, and spiritual
emptiness.
Hypothesis 2: The MSES and its subscales’ scores will demonstrate in-
ternal consistency reliability.
Hypothesis 3: As evidence of convergent validity, the scores on the MSES
and its subscales will exhibit a positive correlation with the SPS (Cull &
924 The Counseling Psychologist 50(7)

Gill, 2002) and negatively correlate with the AHS (Snyder et al., 1991) and
the MLQ-PS (Steger et al., 2006).

Method
Participants
In the first study pool, participants were 541 college students (321 women,
59.3%; 220 men, 40.7%) at a medium-sized four-year university in the Central
Southern region of the United States. Participants ranged in age from 18 to 59
years old with a mean age of 22.82 years (SD = 6.63). In terms of race,
participants identified as Latinx (n = 241; 44.5%), White (n = 196; 36.2%),
Asian American (n = 28; 5.2%), African American (n = 26; 4.8%), Native
American (n = 6; 1.1%), Pacific Islander (n = 3; 0.6%), along with biracial (n =
34; 6.3%) and other (n = 7; 1.3%). In addition to the demographic information,
participants’ suicide attempt history was gathered to further describe the study
sample. Of 541 participants, 83 (15.3%) reported a history of suicide attempts
ranging from one attempt to eight attempts.
In the second study pool, participants were 212 college students (124
women, 58.5%; 88 men, 41.5%) with an age range from 18 to 49 years old
with a mean age of 21.52 years (SD = 5.64). In terms of race, participants
identified as White (n = 85; 40.1%), Latinx (n = 73; 34.4%), Asian American
(n = 17; 8.1%), African American (n = 14; 6.6%), Native American (n = 2; 0.9
%), Pacific Islander (n = 2; 0.9%) along with biracial (n = 13; 6.1%) and other
(n = 6; 2.8%). In addition, 38 of 212 participants (17.9%) reported a history of
suicide attempts ranging from one to three attempts.

Measures
Suicide Probability Scale. The SPS (Cull & Gill, 2002) is a 36-item self-report
measure designed to assess suicide risk in adolescents and adults. Individuals
are asked to rate the frequency of their subjective experience and past be-
haviors using a 4-point Likert-type scale ranging from 1 (none or little of the
time) to 4 (most or all of the time). Responses are summed to evaluate both
general and specific suicide risk along with four key dimensions, including
suicidal ideation, hopelessness, negative self-evaluation, and hostility. Cull
and Gill (2002) provided support for the concurrent validity of the SPS with
the Minnesota Multiphasic Personality Inventory (MMPI) and evidence for
the criterion-related validity demonstrated by the scale’s ability to discrim-
inate between criterion groups of nonclinical participants, psychiatric inpa-
tients, and individuals with a history of a suicide attempt. Cull and Gill also
reported a high test–retest reliability (α = .92 over a 3-week period and α = .94
Ermis-Demirtas et al. 925

over a 10-day period) and internal consistency reliability for the full scale (α =
.93). Similarly, our sample’s SPS ratings yielded an alpha coefficient in the
excellent range (α = .91). The internal consistency coefficients for the sub-
scales were .62 for Negative Self-Evaluation, .78 for Hostility, .80 for
Hopelessness, and .89 for Suicide Ideation (Cull & Gill, 2002).

Adult Hope Scale. Researchers developed the AHS (Snyder et al., 1991) to
assess individuals’ level of hope based on goal-oriented thinking, including
two dimensions: (a) Pathways and (b) Agency. Each subscale consists of four
items with four remaining fillers. This 12-item instrument utilizes an 8-point
Likert-type scale, having responses ranging from 1 (definitely false) to 8
(definitely true). This measure yields a minimum score of 8 and a maximum
score of 64, with higher scores representing a greater level of hope. As
evidence for the convergent validity, the AHS has been correlated positively
with the Life Orientation Test (r = .60), the Self Esteem Scale (r = .45), and
Positive Affect Schedule (r = .53) and correlated negatively with several
scales measures concepts that are antithetical to hope, including hopelessness,
depression, and suicidal ideation (Edwards et al., 2007). In college samples,
the test–retest reliability of the AHS was .85 over a 3-week period. The overall
AHS has demonstrated sound internal reliability, with Cronbach alphas
ranging from .74 to .88 (Snyder et al., 1991). Both subscales have shown
adequate internal reliability. Cronbach alphas have ranged from .70 to .84 for
the Agency subscale and from .63 to .86 for the Pathways subscale (Snyder
et al., 1991). In this study, the Cronbach’s alpha coefficients were .81 for the
Pathways subscale, .79 for the Agency subscale, and .83 for the overall scale.

Meaning in Life Questionnaire–Presence Subscale. The MLQ-PS (Steger et al.,

2006) is a 5-item, self-report inventory developed to measure the extent to
which individuals feel that their lives are significant, are purposeful, and
subjectively make sense to them (Steger et al., 2006). Sample items from the
MLQ-PS include “My life has a clear sense of purpose” and “I have a good
sense of what makes my life meaningful.” Participants use a 7-point Likert-
type scale ranging from 1 (absolutely true) to 7 (absolutely untrue). Users
calculate a total score of the MLQ-PS by summing all items in the inventory
with higher scores indicating a high level of meaning in life. The MLQ-PS was
positively correlated with the measures assessing life satisfaction, positive
emotions, intrinsic religiosity, extraversion and negatively correlated with
depression, negative emotions, and neuroticism (Steger et al., 2006). The
MLQ-PS has good internal consistency (α = .82 to .86) and test–retest re-
liability over a 1-month interval (α = .70). In the current study, the Cronbach
alpha was .95 for the MLQ-PS.
926 The Counseling Psychologist 50(7)

Procedure
Initial Item-Pool Development. Following best practices for item generation in
scale development (Worthington & Whittaker, 2006), we used a four-step
sequential mixed-method approach (Ermis-Demirtas, 2018) to generate an
initial item pool and to establish content-related evidence for the instrument.
These four phases included (a) creation of an initial draft of the MSES, (b)
selection of a panel of experts to evaluate the MSES, (c) a qualitative review of
the MSES, and (d) a quantitative review of the MSES (Ermis-Demirtas, 2018).
We conducted a thorough review of the professional literature on emptiness to
identify components of this construct and develop the initial item pool during
the first step. This in-depth and comprehensive review and analysis of the extant
literature included theoretical, empirical, and clinical issues on emptiness, such
as conceptual articles, qualitative and quantitative studies, clinical notes, and
preexisting measures of emptiness. Using Frankl’s (1984) perspective as the
theoretical base, we defined emptiness as a multidimensional construct, en-
compassing emotional, existential, relational, and spiritual domains, and
identified core features representing these domains. Using the guidelines for
optimal item construction (Kline, 2005), we translated the essential markers of
emptiness into questionnaire statements (e.g., “There is a great void in my life
which I cannot stand.”) and generated an initial pool of 57 items that spread
across the emptiness spheres, including (a) the sense of inner emptiness, (b) the
sense of meaninglessness, (c) the sense of absence of relatedness, and (d) the
sense of spiritual emptiness. As suggested by Józsa and Morgan (2017), we also
included a few reversed items considering the advantages, such as encouraging
participants to read all the items carefully and improving the instrument validity.
After constructing the initial item pool, we set out to create the format for
the instrument. We selected a 5-point Likert-type scale response set with
values ranging from 1 (none or little of the time true of me) to 5 (most or all of
the time true of me) as a combination of reflection and frequency response
anchors to allow individuals to express how much they agree or disagree with
a particular statement (Wu & Leung, 2017). We also avoided using the
midpoint categories such as “I do not know” or “undecided” to obtain higher
data accuracy and forgo participant indecisiveness (DeVellis, 2016). Direc-
tions for the instrument instructed participants to determine how often each
statement describing their feeling, perception, or behavior was true for them
over the past year based on their subjective experiences.
Our second step was to identify and contact content experts and request them
to review the instrument’s initial draft. We contacted a total of 15 experts, eight of
them volunteered to serve as a reviewer. Out of eight content experts, we recruited
six from counseling, one from clinical psychology, and one from psychiatry. Each
panel member had an established reputation as a scholar in their field, worked as a
full-time professor, and had clinical experience ranging from approximately 15 to
Ermis-Demirtas et al. 927

30 years. Further, six out of eight reviewers had experience with instrument
development germane to mental health. Regarding the specialization of the
content reviewers in emptiness, two experts were identified to have either doctoral
dissertations or peer-reviewed articles in international journals related to the topic.
For the third phase, we dispatched a qualitative assessment packet, in-
cluding the instrument and the instructions. We requested experts to provide
feedback on the instrument’s content, organization, existing questions, and
administration process. The review of the experts’ feedback resulted in re-
vising 23 items, adding four items, and dropping nine items from the in-
strument draft (52 items in total). During the last phase, we shared a
quantitative assessment packet with the reviewers and requested them to rate
each item’s appropriateness for the instrument based on a 3-point response set
with 1 indicating essential, 2 useful but not essential, and 3 not necessary. We
employed Lawshe’s (1975) technique as a relatively straightforward quan-
titative approach to content validity (Ermis-Demirtas, 2018). After receiving
the experts’ ratings, we calculated the content validity ratio (CVR) value for
each item using the CVR formula (Ne N/2)/(N/2), in which the Ne is the
number of reviewers indicating essential, and N is the total number of re-
viewers. Using the Lawshe’s table (1975), we determined the CVRcritical value
to be .75 since the judging panel constituted eight content reviewers. The
items with the CVR values exceeding the CVRcritical value of .75 remained in
the final instrument. The last step in establishing content-related evidence was
to compute the CVI value within the retained 39 items. Researchers found that
the 39-item instrument’s CVI value was .89, considered adequate based on
recommendations (CVI > .80; Ermis-Demirtas, 2018).

Participant Recruitment. Participants for this study were recruited from un-
dergraduate courses offered in a range of majors at a Hispanic-serving in-
stitution in the Central Southern United States. Upon obtaining permission
from the department chairs, program coordinators, and course instructors, the
primary researcher in this study attended classes and presented the study to the
potential participants. Volunteer students willing to participate in this research
were disseminated a survey packet, including the MSES, a demographic
questionnaire, three other instruments, and provided time to complete the
surveys in class.

Data Analysis
Statistical Power Analysis. We conducted a statistical power analysis to identify
the required sample size for running factor analysis based on a ratio of 10
participants-to-item, n/p ≥ 10 (Mvududu & Sink, 2013). Further, scholars
suggested a sample size greater than 300 for performing EFA and 200 for CFA
as appropriate (Brown, 2015; Tabachnick & Fidell (2019). Following these
928 The Counseling Psychologist 50(7)

guidelines, the sample sizes are considered adequate to conduct EFA (n = 541,
p = 39 > 13) and CFA (n = 212, p = 13 > 16).

Preliminary Analysis. We followed two steps to clean the data. First, we

screened the data for missing cases and determined that the data were missing
at random and ignorable (e.g., less than 5%; Brown, 2015). Because of the
minimal amount of missing data, we determined listwise deletion at the first
step, following the best practices to conduct EFA and CFA. This process
resulted in removing 27 cases (3.59%) from the initial data (n = 780) due to the
extensive participant noncompletion (more than 20% of the items were un-
answered; Parent, 2013). The remaining missing cases constituted 0.13% of
the overall data and 0.2% of the MSES (n = 753; less than 5%; Eekhout et al.,
2014). The MCAR test was nonsignificant (p = .807). Thus, the next step was
to impute this small percentage of missing values using the series mean
function in SPSS.
Next, we assessed intercorrelations among 39 items to determine whether
any items in the scale were highly correlated with one another to eliminate
redundancy (Watson, 2017). We made the decision to delete the items using
two well-recognized criteria. First, we used the correlation matrix among
items to check the pattern of relationships. The intercorrelation matrix for
most items was greater than .2, indicating reasonable factorability (Watson,
2017). Second, we checked the correlation coefficients between items and the
correlation matrix’s determinant to test multicollinearity. Most of the corre-
lation coefficients were lower than .9, and the determinant value was .002,
suggesting that multicollinearity was not an issue (Tabachnick & Fidell,
2019). Furthermore, we tested data for univariate and multivariate normality
using the critical cut-off value of 2.0 for skewness and 7.0 for kurtosis (Brown,
2015). All items had values less than 2.0 for skewness and kurtosis, indicating
that the data were normally distributed.

Results
Evidence Regarding Internal Structure
Exploratory Factor Analysis. Following the methodological literature, we first
analyzed data from 541 college students using the Kaiser-Meyer-Olkin
(KMO) measure of sampling adequacy and Bartlett’s test of sphericity to
determine the suitability of the sample for factor analysis. Results suggested a
meritorious level of homogeneity (KMO = .96) and were significant for
Bartlett’s test of sphericity, χ 2(n = 541, 190) = 18075.255, p < .001, indicating
the data appropriate for factor analysis procedures. Once data were deemed
appropriate for analysis, the PAF procedure with a Promax rotation was
applied to extract factors from the data (Mvududu & Sink, 2013; Tabachnick
Ermis-Demirtas et al. 929

& Fidell, 2019). To determine the exact number of factors to extract, we used
the following criteria: (a) factors with eigenvalues greater than 1.0, (b) visual
analysis of the scree plot, (c) parallel analysis, and (d) the conceptual
meaningfulness of the factors (Watson, 2017). Using the extraction criteria,
we initially retained four factors that accounted for 75.4% of the variance for
the entire set of variables. The scree plot test also confirmed the four-factor
solution. In addition to PAF and scree plot, we also conducted a parallel
analysis in which the eigenvalues for the actual data were compared to ei-
genvalues generated from a random data set of like size (Watson, 2017). We
utilized the rawpar.script to run parallel analysis in SPSS using PAF for
normally distributed random data generation with a 95% confidence level.
Four factors were significant because the eigenvalues generated from the
actual data were greater than the eigenvalues from the random data. Noticing
consistent results across the methods, we decided to retain four factors
initially.
Next, we decided to exclude 16 items with less than .50 factor coefficients
(Mvududu & Sink, 2013) and eight items with more than .32 cross-loadings
using the item retention criteria (Tabachnick & Fidell, 2019; Watson, 2017).
Two items that loaded on a single factor did not fit with the factorial solution
after rerunning the EFA. Therefore, we decided to drop these items from
further analysis considering Watson’s recommendations. This procedure re-
sulted in a three-factor solution with 13 items accounting for 82.82% of the
variance in the MSES items. Consequently, we ended up loading all items on
said dimensions. Factor 1, labeled Sense of Inner Emptiness (SIE), consisted
of five items (Items 1, 2, 6, 7, and 8) and accounted for approximately 66.48%
of the variance. The second factor, named Sense of Absence of Relatedness
(SAR), included five items (Items 19, 20, 26, 27, and 28) and explained 8.7%
of the variance. Lastly, the third factor, labeled Sense of Meaninglessness
(SM), contained three items (Items 10, 11, and 13) and accounted for 7.61% of
the variance. We present the three factors and their respective items, and factor
loadings in Table 1.

Confirmatory Factor Analysis. After EFA, we conducted a series of CFA in the

second sample (n = 212) to verify the number of underlying latent variables
and the pattern of observed variable-factor relationship (Brown, 2015). Due to
the intercorrelations among three factors extracted from EFA (r > .33; Brown,
2015), we evaluated three primary models to examine the underlying structure
of emptiness and compare the model fit. These structures included (a) a three-
factor first-order correlated model corresponding to the SIE, SAR, and SM
(Figure 1, Model A), (b) a higher-order three-factor model corresponding to
the SIE, SAR, and SM as the first-order factors and the Sense of Emptiness as
the second-order factor (Figure 1, Model B), and (c) a bifactor model.
930 The Counseling Psychologist 50(7)

To measure the degree of fit, we used fit indices including the chi-square

statistics, standardized root mean residual (SRMR), goodness of fit index
(GFI), comparative fit index (CFI), Tucker-Lewis index (TLI), and the root
mean square error of approximation (RMSEA) metrics of model fit. To
evaluate and compare model fit, we used the following cut-off criteria for a
strong model fit, including χ 2/df < 2, SRMR < .05, GFI > .95, CFI > .95, TLI >
.95, and RMSEA < .06 and the benchmarks for an adequate model fit, in-
cluding χ 2/df < 5, SRMR < .08, GFI > .90, CFI > .90, TLI > .90, and RMSEA
< .08 (Brown, 2015).
Three-Factor Correlated Model. The three-factor first-order correlated
model with factors corresponding to SIE, SAR, and SM evidenced acceptable
model fit (see Table 2). All factor loadings of this model were significant
ranging from .86 to .97. The correlations among the three factors in this model
were also all significant and as follows: SIE with SAR (r = .71), SAR with SM
(r = .65), and SIE with SM (r = 77).
Higher-Order Three-Factor Model. We tested a higher-order model with
three factors corresponding to the SIE, SAR, and SM as the first-order factors

Table 1. Pattern Matrix for the Three-Factor Solution

Items F1 F2 F3 CFA M SD

1. I feel empty inside. .94 .89 2.14 1.36

7. I crave for something to fill me up .90 .86 2.25 1.41
emotionally.
6. I feel emotionally hungry. .89 .87 2.12 1.34
8. It feels like there is a great void inside me. .86 .89 1.92 1.32
2. I feel as if a part of me is missing. .82 .87 2.05 1.34
19. I am unable to relate to significant others. .97 .75 1.76 1.19
27. I do not have a sense of connectedness .92 .75 1.95 1.34
with society.
26. I feel I do not belong to any group .88 .77 1.95 1.33
around me.
28. I feel detached from the shared .87 .81 1.90 1.31
meaning in my community.
24. I feel distant from people around me. .83 .74 1.75 1.15
10. I cannot find meaning in my life. .86 .85 1.73 1.21
13. My life has no clear direction. .82 .80 1.88 1.24
11. There is no clear purpose of my life. .81 .85 1.78 1.22
Explained Variance (%) 66.48 8.73 7.61
Cronbach’s alpha (α) .98 .96 .97
Note. n = 541. F1 = Sense of Inner Emptiness exploratory factor analysis (EFA) factor loadings; F2 =
Sense of Absence of Relatedness EFA factor loadings; F3 = Sense of Meaninglessness EFA factor
loadings. CFA = confirmatory factor analysis item factor loadings.
Ermis-Demirtas et al. 931

Figure 1. Compared Confirmatory Factor Analysis Models of the MSES.

Note. MSES = Multidimensional Sense of Emptiness Scale. Model A = three-factor correlated
model; Model B = higher-order model (three second-order factors and one single, second-order
factor); Model C = bifactor model.

and Sense of Emptiness as a single, second-order factor. CFA procedures

confirmed this internal structural robustness by demonstrating an excellent fit
of the data to the model (see Table 2). This model’s factor loadings were
significant, ranging from .74 to .89 for the three first-order factors (see Table 1)
and .77 to .91 for the single, second-order factor.
Bifactor Model. The bifactor model generated a poor model fit (see Table
2).
Model Comparisons. Among the three models we tested, the higher-order
model produced a good fit, the three-factor correlated model produced an
acceptable fit, and the bifactor model generated a poor model fit. We compared
the higher-order model with the three-factor correlated model. A significant
chi-square difference, Δχ 2(2) = 16.26, p < .001, indicated that the higher-order
model was better. We also examined the Akaike Information Criteria (AIC;
Akaike, 1973). A smaller AIC (189.63) for the higher-order model indicated
that it was better than the three-factor correlated model (AIC = 285.03) and the
932 The Counseling Psychologist 50(7)

Table 2. Fit Indexes from the Confirmatory Factor Analyses with the MSES Data

RMSEA
Model χ 2/df SRMR CFI GFI TLI 90% CI [LL, UL] AIC

3-Factor Correlated Model 2.13 .05 .96 .91 .95 .06 [.06, .08] 285.03
Higher-Order Model 1.93 .04 .99 .96 .99 .05 [.04, .05] 189.63
Bifactor Model 3.98 .08 .94 .88 .93 .09 [.08, .11] 359.30
Note. N = 212; MSES = Multidimensional Sense of Emptiness Scale; SRMR = Standardized Root-
Mean-Square Residual; CFI = comparative fit index; GFI = goodness of fit index; TLI = Tucker-
Lewis index; RMSEA = root-mean-square error of approximation; CI = confidence interval; LL =
lower limit; UL = upper limit; AIC = Akaike information criteria.

bifactor model (AIC = 359.30). These comparison results suggest that the
higher-order model appeared to best represent the underlying structure of the
emptiness items.

Evidence Regarding Internal Consistency Reliability

We calculated internal consistency reliability coefficients for the total scale
and three subscales using Cronbach’s alpha with an α of .70 or higher as a
standard (DeVellis, 2016). The reliability estimate for the 13-item total scale
was .97 (n = 753), indicating strong reliability. The reliability estimates for the
subscales were .98 for the five-item SIE, .96 for the five-item SAR, and .97 for
the three-item SM. These values also indicated high internal consistency
reliability evidence for scores obtained from these three subscales (α > .70).

Evidence Regarding Relations to Other Variables

Regarding evidence for convergent validity, bivariate correlation analysis of
scores on the MSES and its subscales with the SPS revealed a significant
positive relationship (see Table 3), indicative of large effect size (Swank &
Mullen, 2017). As an additional source of evidence for convergent validity, a
negative relationship of the MSES and its subscales was identified with the
AHS and the MLQ-PS (see Table 3), indicative of large effect size (Swank &
Mullen, 2017). The strong relationships of the MSES and its subscales with
other instruments measuring related constructs provide preliminary evidence
confirming convergent validity.

Discussion
Although historically, emptiness has received little consideration in the
counseling literature relative to other types of negative experience, there is
growing interest in studying emptiness, especially concerning suicide risk.
Ermis-Demirtas et al. 933

Table 3. Descriptive Statistics and Bivariate Correlations for Scores on the MSES and
its Components

Scales and
Subscales M SD 1 2 3 4 5 6 7

1. MSES–Total 25.18 14.29 1.00

2. MSES–SIE 10.49 6.43 .82* 1.00
3. MSES–SM 5.41 3.63 .67* .64* 1.00
4. MSES–SAR 9.30 5.92 .71* .58* .52* 1.00
5. SPS 67.40 22.79 .83* .86* .79* .81* 1.00
6. AHS 41.27 9.18 .64* .62* .57* .51* .73* 1.00
7. MLQ–PS 24.82 7.41 .72* .70* .78* .67* .69* .74* 1.00

Note. MSES = Multidimensional Sense of Emptiness Scale. SIE = Sense of Inner Emptiness Subscale;
SM = Sense of Meaninglessness Subscale; SAR = Sense of Absence of Relatedness Subscale; SPS =
Suicide Probability Scale; AHS = Adult Hope Scale; MLQ-PS = Meaning in Life Questionnaire–
Presence Subscale.
*p < .05.

Perhaps more crucially, there is an increasing need to understand this

prevalent and often detrimental experience. Moreover, the evaluation of the
appropriate factor structure of emptiness has critical assessment and con-
ceptual implications. In light of growing need and interest in studying
emptiness, applying Frankl’s conceptualization as a theoretical base, we
aimed to develop and validate a new instrument, the MSES. Using a more
practitioner-friendly vocabulary, we designed the MSES to measure empti-
ness in the general population for counseling research and practice purposes.
To achieve this goal, we first explored the factorial structure of the emerging
instrument within a sample of 541 undergraduate students. The EFA resulted
in the emergence of three distinct but significantly correlated factors with 13
items: (a) SIE, (b) SAR, and (c) SM. These conceptually meaningful factors
accounted for 82.8% of the variance. Although we anticipated a four-factor
structure with four corresponding subscales, including the Sense of Spiritual
Emptiness as an additional separate dimension, the EFA solution did not
support this conceptual structure. Frankl’s conceptualization of emptiness
focused on the intrapersonal and existential aspects of this concept yet in-
volved the interpersonal domain of emptiness. As an additional sphere,
spiritual emptiness was included in the theoretical framework considering
Dunn’s (1994) qualitative study findings. However, Dunn explored psy-
chological emptiness among religious women, which may have limited the
representation of this experience in the general population and resulted in the
exclusion of spiritual emptiness from the final solution.
We then conducted CFA in an additional sample (n = 212) to test the
emerged three-factorial solution. More specifically, because the three factors
934 The Counseling Psychologist 50(7)

represent specific yet correlated aspects of emptiness, we evaluated three

primary structures, including (a) a three-factor correlated model, (b) a higher-
order three-factor model, and (c) a bifactor model. The study findings showed
that the MSES data are best represented by a higher-order model with three
components corresponding to the SIE, SAR, and SM as the first-order factors
and the sense of emptiness as the single, second-order factor. These results
also indicate that the total MSES score is meaningful. Because the second-
order factor provides a statistically parsimonious explanation for the corre-
lations among the first-order factors, these factors can be considered as
specific components of a single, general construct of emptiness rather than
separate constructs that are correlated with one another. The higher-order
model was also theoretically preferred and chosen as the final structural model
for the MSES.
Taken together, the SIE factor consists of five items. The statements related
to this factor collectively describe the intrapersonal aspect of emptiness, such
as feelings of emotional hunger, a part of self being missing, unbearable inner
void, and emotional yearning to fill that void. The SAR factor also includes
five items. This factor describes the interpersonal manifestations of emptiness
that refer to feeling detached from the people, society, and shared meaning in
the community. The SM factor contains three items. This factor depicts the
existential aspect, encompassing the lack of meaning, purpose, and life di-
rection. For the 13-item MSES, the total score ranges from 13 to 65, with
higher scores indicating greater levels of emptiness. The scores for the
subscales range from 5 to 25 for the SIE and the SAR and 3 to 15 for the SM.
Our next step was to examine the internal consistency reliability coeffi-
cients of the MSES and the emergent factors. Reliability estimates of .97, .98,
.96, and .97 for the MSES as well as for the SIE, SAR, and SM, respectively,
exceeded standard conventions where values above .70 are viewed acceptable
(DeVellis, 2016) and suitable for assessment applications (Bardhoshi &
Erford, 2017). These reliability estimates on the MSES and its factors of-
fer a degree of consistency and precision, which might support its use for
reliable assessment of emptiness in both counseling research and practice.
For our final step in the validation process, we examined convergent
validity by correlating scores on the MSES and its subscales with the existing
instruments designed to assess similar constructs. Participant scores on the
MSES and its subscales correlated with the Suicide Probability Scale (Cull &
Gill, 2002). This outcome seems consistent with the previous findings, in-
dicating emptiness as a significant marker of suicidal outcomes (Blascallo-
Fontecilla et al., 2013; Segal-Engelchin et al., 2015). To provide further
convergent validity evidence, we tested the relationship of the MSES and its
subscales with the existing measures of hope and presence of meaning in life.
As predicted, the MSES total score and its subscale scores were significantly
correlated with the AHS (Snyder et al., 1991) and the MLQ-PS (Steger et al.,
Ermis-Demirtas et al. 935

2006), which seems consistent with the literature (Cheng et al., 2013; Frankl,
1984). These results altogether provide support for the convergent validity of
the MSES.

Limitations
There are a few limitations of the MSES that readers should note. First, we
conducted this scale development and validation study to measure emptiness
with a young, fairly educated adult sample, which limits the representation of
the general population. Thus, researchers should be cautious when using the
MSES in populations with different characteristics. Moreover, most partic-
ipants were self-identified as Latinx and White, limiting the study’s gener-
alizability across various ethnicities and cultural backgrounds. Future research
could examine the MSES’s validity in different samples, including ethnically
diverse and clinical populations. Further, this study was an initial investigation
of the MSES’s psychometric properties; therefore, the findings merit further
inquiry. In assessing various psychological emptiness domains, initial items
were developed based on a four-dimensional conceptualization of this con-
struct, which might constitute another limitation. Besides, the MSES items
excluded from the final instrument may also represent emptiness areas that
cannot be measured by items retained within the assessment. Lastly, many
empirical studies that rely on self-report measures are prone to the social
desirability bias, and this study was no exception.
Future researchers may conduct follow-up studies addressing the limita-
tions noted for this investigation, while also expanding on this study’s
findings. Future research is needed to conduct a CFA analysis to test the
higher-order structure of the MSES using various samples, including a clinical
sample. The study could also be replicated to include a wider variety of
individuals to address sampling limitations, as the factorial structure of this
instrument was analyzed based on data obtained from a single sample of
young, fairly educated adults. Future researchers may include an increased
representation of diversity, including different ethnic identities and cultural
backgrounds. Further research could expand the study by assessing the
between-group factor structure of the MSES via measurement invariance
testing to determine whether the MSES performs consistently across different
samples (e.g., gender or mental disorder groups). Researchers might further
explore the psychometric properties of the MSES to provide additional re-
liability and validity evidence for this measure. Test–retest reliability mea-
sures would be useful for further investigations of convergent validity with
existing measures of emptiness and associated mental disorders, including
depression and borderline personality disorder.
936 The Counseling Psychologist 50(7)

Implications for Practice, Advocacy, Education, Training,

and Research
Despite the limitations of the MSES, the results derived from our psycho-
metric evaluation can have implications for counseling practice, research,
advocacy, and training. As a comprehensive measure of emptiness with
promising validity and reliability evidence developed in a relatively larger
sample with a reasonable diversity, counseling practitioners might use the 13-
item three-factor MSES for assessment and screening purposes. Clinicians
and trainees may consider using the subscales or the total score, given the-
oretical reasoning and empirical evidence, suggesting the higher-order model
fits well with the MSES data. The use of the MSES might allow clinicians to
conduct a comprehensive assessment and gain additional information about
their clients (Peteet, 2011). Such information would be useful for counseling
professionals to target the problem areas based on the subscales with the
highest scores. Furthermore, the use of the MSES might help counselors to
gain insight into the client’s perception of emptiness. The counselor–client
interpretation of the scores can lead to rich dialogue in which both may gain
insight into the client’s experiences. The results of this study were preliminary,
pertinent to the psychometric evidence of the MSES, and this scale was
developed within the general population. Therefore, we suggest using the
MSES with caution in a clinical population as further evidence is needed to
replicate this study on a sample of clients.
For the use of MSES in counseling research, the 13-item psychometrically
sound, user-friendly, and theory-driven instrument might provide researchers
with a promising assessment tool to measure emptiness and investigate the
relationship with other psychological constructs. For example, the strong
positive relationship identified between the MSES and the SPS might provide
a basis to examine further the relationship between emptiness and suicidal
ideation and behavior. Concerning the MSES scoring, researchers are advised
to compute both the subscale scores and the total score for the interpretation
purposes. The three-factor higher-order model’s validity tests suggest that the
MSES subscales should not be avoided because they may provide additional
information beyond the total score.
Lastly, as advocates for best practices in counseling, counselor educators
and supervisors may help integrate the concept of emptiness into the
counseling training programs to prepare better emerging professionals to work
with clients experiencing emptiness. This could reduce the ambiguity in this
area and enhance students’ confidence to work with such clients. Such in-
tegration may be achieved via case studies in which hypothetical clients with
emptiness and a score on the MSES could be given to students to work on case
conceptualization. Counseling students could also be educated on the MSES
to use it for counseling screening and assessment purposes.
Ermis-Demirtas et al. 937

Taken together and given the growing body of literature linking emptiness
with a range of mental health issues, the potential applications of an as-
sessment tool measuring emptiness is evident. Thus, a well-grounded and
psychometrically sound measure of emptiness will better allow for empirical
investigations into the possible risk factors and outcomes associated with
emptiness. This study involved the psychometric evaluation of the MSES as a
theoretically and empirically grounded and comprehensive measure of
emptiness, including such processes as writing and reviewing items, analyzing
the factorial structure, evaluating the reliability of the MSES, and establishing
convergent validity evidence. Although the validity and reliability evidence
are preliminary, the conceptualization of emptiness, the development of
MSES, and the conceptual clarity achieved through this study represent a
valuable contribution to the counseling field. As a multidimensional and well-
validated measure of emptiness, we hope that the MSES will serve as the basis
for self-report assessment of emptiness and give rise to many useful appli-
cations in counseling research and practice, moving beyond the existing
measures’ current limitations in content coverage and scale validation
procedures.

Author Note
This study was funded by the Texas Counseling Association. We have no conflicts of
interest to disclose.

ORCID iD
Hulya Ermis-Demirtas  https://orcid.org/0000-0001-7247-5038

References
Akaike, H. (1973). Information theory and an extension of the maximum likelihood
principle. In B. N. Petrov & F. Csaki (Eds.), Second international symposium on
information theory (pp. 267–281). Akademiai Kiado.
American Psychiatric Association. (2013). Diagnostic and statistical manual of mental
disorders (5th ed.). https://doi.org/10.1176/appi.books.9780890425596
Arndt, B., Gunther, A., Bauman, K., Frick, E., & Jacobs, C. (2013). Spiritual dryness
as a measure of a specific spiritual crisis in Catholic priests: Associations with
symptoms of burnout and distress. Evidence-Based Complementary and Alter-
native Medicine, 2013, 246797. https://doi.org/10.1155/2013/246797
Bardhoshi, G., & Erford, B. T. (2017). Process and procedures for estimating score
reliability and precision. Measurement and Evaluation in Counseling & De-
velopment, 50(4), 256–263. https://doi.org/10.1080/07481756.2017.1388680
Blasco-Fontecilla, H., Leon-Martinez, V., Delgado-Gomez, D., Giner, L., Guillaume,
S., & Courtet, P. (2013). Emptiness and suicidal behavior: An exploratory review.
Suicidology Online, 4, 21–32.
938 The Counseling Psychologist 50(7)

Brown, T. A. (2015). Confirmatory factor analysis for applied research (2nd ed.).
Guilford.
Buggs, G. R. (1996). Emptiness: Assessment, origins, sequelae, and relationship to
abuse (Publication No. 9704787) [Doctoral dissertation. California Institute of
Integral Studies]. ProQuest Dissertations & Thesis Global.
Cheng, T. K., Ho, D. Y. F., Xie, W., Wong, H. Y. K., & Cheng-Lai, A. (2013).
Alienation, despair and hope as predictors of health, coping and non-engagement
among nonengaged youth: Manifestations of spiritual emptiness. Asia Pacific
Journal of Counselling and Psychotherapy, 4(1), 18–30. https://doi.org/10.1080/
21507686.2012.756405
Chia, B. H., Chia, A., & Tai, B. C. (2008). Suicide letters in Singapore. Archives of
Suicide Research, 12(1), 74–81. https://doi.org/10.1080/13811110701801069
Costello, C. G., & Comrey, A. L. (1967). Scales for measuring anxiety and depression.
Journal of Psychology, 66, 303–313.
Cull, J. G., & Gill, W. S. (2002). Suicide Probability Scale [SPS] manual. Western
Psychological Services.
Cushman, P. (1990). Why the self is empty. American Psychologist, 45(5), 599–611.
https://doi.org/10.1037/0003-066X.45.5.599
DeVellis, R. F. (2016). Scale development: Theory and applications (4th ed.). Sage.
Drapeau, C. W., & McIntosh, J. L. (2020). U.S.A. suicide 2019: Official final data.
American Association of Suicidiology. https://suicidology.org/wp-content/
uploads/2021/01/2019datapgsv2b.pdf
Dunn, S. R. (1994). The phenomenon of psychological emptiness in the lives of re-
ligious women between the ages of forty through sixty-five (Publication No.
9320945) [Doctoral dissertation, The Fielding Institute]. ProQuest Dissertations
& Theses Global.
Edwards, L. M, Rand, K. L., Lopez, S. J., & Snyder, C. R. (2007). Understanding hope:
A review of measurement and construct validity research. In A. D. Ong &
M. H. M. van Dulmen (Eds.), Handbook of methods in positive psychology (pp.
83–95). Oxford University.
Eekhout, I., de Vet, H. C. W., Twisk, J. W. R., Brand, J. P. L., de Boer, M. R., &
Heymans, M. W. (2014). Missing data in a multi-item instrument were best
handled by multiple imputation at the item score level. Journal of Clinical
Epidemiology, 67(3), 335–342, https://doi.org/10.1016/j.jclinepi.2013.09.009
Elsner, D., Broadbear, J. H., & Rao, S. (2018). What is the clinical significance of
chronic emptiness in borderline personality disorder? Australian Psychiatry,
26(1), 88–91. https://doi.org/10.1177/1039856217734674
Ermis-Demirtas, H. (2018). Establishing content-related evidence for assessment in
counseling: Application of a sequential mixed-method approach. International
Journal for the Advancement of Counseling, 40(4), 387–397. https://doi.org/10.
1007/s10447-018-9332-4
Frankl, V. (1969). The will to meaning: Foundations and applications of Logotherapy.
World Publishing Company.
Ermis-Demirtas et al. 939

Frankl, V. (1984). Man’s search for meaning. Simon and Schuster.

Hazell, C. H. (1984). A scale for measuring experienced levels of emptiness and
existential concern. The Journal of Psychology, 117(2), 177–182, https://doi.org/
10.1080/00223980.1984.9923674
Hazell, C. H. (2003). The experience of emptiness. 1st Books Library.
Józsa, K., & Morgan, G. A. (2017). Reversed items in Likert scales: Filtering out invalid
responders. Journal of Psychological and Educational Research, 25(1), 7–25.
Kline, T. J. B. (2005). Psychological testing: A practical approach to design and
evaluation. Sage. https://doi.org/10.4135/9781483385693
Klonsky, E. D. (2008). What is emptiness? Clarifying the 7th criterion for borderline
personality disorder. Journal of Personality Disorders, 22(4), 418–426. https://
doi.org/10.1521/pedi.2008.22.4.418
Lawshe, C. H. (1975). A quantitative approach to content validity. Personnel Psy-
chology, 28(4), 563–575. https://doi.org/10.111/j.1744-6570.1975.tb01393.x
Meehan, S. D. (2007). Emptiness and its role in disordered eating (Publication No.
3265859) [Doctoral dissertation. Massachusetts School of Professional Psy-
chology]. ProQuest Dissertations & Theses Global.
Miller, C. E., Townsend, M. L., Day, N. J. S., & Grenyer, B. F. S. (2020). Measuring the
shadows: A systematic review of chronic emptiness in borderline personality
disorder. PLoS One, 15(7), 1–49. https://doi.org/10.1371/journal.pone.0233970
Mvududu, N. H., & Sink, C. A. (2013). Factor analysis in counseling research and
practice. Counseling Outcome Research and Evaluation, 4(2), 75–98. https://doi.
org/10.1177/2150137813494766
Parent, M. C. (2013). Handling item-level missing data: Simpler is just as good. The
Counseling Psychologist, 41(4), 568–600. https://doi.org/10.1177/
0011000012445176
Parkes, C. P. (2001). Bereavement: Studies of grief in adult life. Taylor and Francis.
Peteet, J. R. (2011). Approaching emptiness: Subjective, objective, and existential
dimensions. Journal of Religion and Health, 50(3), 558–563. https://doi.org/10.
1007/s10943-010-9443-7
Rallis, B. A., Deming, C. A., Glenn, J. J., & Nock, M. K. (2012). What is the role of
dissociation and emptiness in the occurrence of nonsuicidal self-injury? Journal
of Cognitive Psychotherapy, 26(4), 287–298. https://doi.org/10.1891/0889-8391.
26.4.287
Roos, C. R., Kirouac, M., Pearson, M. R., Fink, B. C., & Witkiewitz, K. (2015).
Examining temptation to drink from an existential perspective: Associations
among temptation. Purpose in Life, and Drinking Outcomes, 29(3), 716–724.
https://doi.org/10.1037/adb0000063
Segal-Engelchin, D., Kfir-Levin, N., Neustaedter, S. B., & Mirsky, J. (2015). Mental
pain among female suicide attempt survivors in Israel: An exploratory qualitative
study. International Journal of Mental Health Addiction, 13(4), 423–434, https://
doi.org/10.1007/s11469-015-9545-2
940 The Counseling Psychologist 50(7)

Snyder, C. R., Harris, C., Anderson, J. R., Holleran, S. A., Irving, L. M., Sigmon, S. T.,
Yoshinobu, L., Gibb, J., Langelle, C., & Harney, P. (1991). The will and the ways:
Development and validation of an individual-differences measure of hope.
Journal of Personality and Social Psychology, 60(4), 570–585. https://doi.org/10.
1037/0022-3514.60.4.570
Steger, M. F., Frazier, P., Oishi, P., & Kaler, M. (2006). The Meaning in Life Ques-
tionnaire: Assessing the presence of and search for meaning in life. Journal of
Counseling Psychology, 53(1), 80–93. https://doi.org/10.1037/0022-0167.53.1.80
Swank, J. M., & Mullen, P. R. (2017). Evaluating evidence for conceptually related
constructs using bivariate correlations. Measurement and Evaluation in Coun-
seling & Development, 50(4), 270–274. https://doi.org/10.1080/07481756.2017.
1339562
Tabachnick, B. G., & Fidell, L. S. (2019). Using multivariate statistics (7th ed.).
Pearson.
Watson, J. C. (2017). Establishing evidence for internal structure using exploratory
factor analysis. Measurement and Evaluation in Counseling & Development,
50(4), 232–238. https://doi.org/10.1080/07481756.2017.1336931
World Health Organization. (2016). International statistical classification of diseases
and related health problems (10th ed.). https://icd.who.int/browse10/2016/en
Worthington, R. L., & Whittaker, T. A. (2006). Scale development research: A content
analysis and recommendations for best practices. The Counseling Psychologist,
34(6), 806–838. https://doi.org/10.1177/0011000006288127
Wu, H., & Leung, S. (2017). Can Likert scales be treated as interval scales? – A
simulation study. Journal of Social Service Research, 43(4), 527–532. http://doi.
org/10.1080/01488376.2017.1329775

Author Biographies
Hulya Ermis-Demirtas, PhD, is an assistant professor in the Department of
Counselor Education at Northeastern Illinois University. Her research agenda
includes instrument development, social determinants of mental health, suicide
risk factors, and counseling children and adolescents. She is a recent recipient of
the North Central Association for Counselor Education and Supervision Re-
search Award and Emerging Leader Award.
Robert L. Smith, PhD, is a professor and coordinator of the doctoral program
in counselor education at Texas A&M University-Corpus Christi. His research
focus includes instrument development, efficacy of self-enhancement pro-
grams for children and adolescents, assessment of online instruction, efficacy
of addictions counseling, and marriage and family therapy interventions. He is
a past president of American Counseling Association, International Asso-
ciation of Marriage and Family Counselors, and National Career Develop-
ment Association.
Ermis-Demirtas et al. 941

Joshua C. Watson, PhD, is a professor and chair in the Department of

Counseling and Educational Psychology at Texas A&M University-Corpus
Christi. His research interests include scale development, cross-cultural
validation of measures, and college student adjustment. He is a past presi-
dent of the Association for Assessment and Research in Counseling and an
American Counseling Association Fellow.