Journal of

Clinical Medicine

Review
Executive Dysfunctions in Schizophrenia: A Critical Review of
Traditional, Ecological, and Virtual Reality Assessments
Ernest Tyburski 1, * , Monika Mak 2 , Andrzej Sokołowski 3 , Anna Starkowska 4 , Ewa Karabanowicz 5 ,
Magdalena Kerestey 5 , Zofia Lebiecka 2 , Joanna Preś 2 , Leszek Sagan 6 , Jerzy Samochowiec 7
and Ashok S. Jansari 8

1 Institute of Psychology, SWPS University of Social Sciences and Humanities, 61-719 Poznań, Poland
2 Department of Health Psychology, Pomeranian Medical University in Szczecin, 71-457 Szczecin, Poland;
[email protected] (M.M.); [email protected] (Z.L.); [email protected] (J.P.)
3 Memory and Aging Center, Department of Neurology, UCSF Weill Institute for Neurosciences, University of
California, 675 Nelson Rising Lane, Suite 190, San Francisco, CA 94143, USA; [email protected]
4 Faculty of Psychology in Wrocław, SWPS University of Social Sciences and Humanities,
53-238 Wrocław, Poland; [email protected]
5 Institute of Psychology, University of Szczecin, 71-017 Szczecin, Poland; [email protected] (E.K.);
[email protected] (M.K.)
6 Department of Neurosurgery, Pomeranian Medical University in Szczecin, 71-252 Szczecin, Poland;
[email protected]



7 Department of Psychiatry, Pomeranian Medical University in Szczecin, 71-457 Szczecin, Poland;


[email protected]
Citation: Tyburski, E.; Mak, M.; 8 Department of Psychology, Goldsmiths, University of London, New Cross, London SE14 6NW, UK;
Sokołowski, A.; Starkowska, A.; [email protected]
Karabanowicz, E.; Kerestey, M.; * Correspondence: [email protected]; Tel.: +48-61-271-12-22
Lebiecka, Z.; Preś, J.; Sagan, L.;
Samochowiec, J.; et al. Executive Abstract: In recent years, interest has grown in measuring executive function in schizophrenia with
Dysfunctions in Schizophrenia: A ecological and virtual reality (VR) tools. However, there is a lack of critical analysis comparing those
Critical Review of Traditional, tools with traditional ones. This paper aims to characterize executive dysfunction in schizophrenia
Ecological, and Virtual Reality by comparing ecological and virtual reality assessments with traditional tools, and to describe the
Assessments. J. Clin. Med. 2021, 10,
neurobiological and psychopathological correlates. The analysis revealed that ecological and VR tests
2782. https://doi.org/10.3390/
have higher levels of verisimilitude and similar levels of veridicality compared to traditional tools. Both
jcm10132782
negative symptoms and disorganization correlate significantly with executive dysfunction as measured
by traditional tools, but their relationships with measures based on ecological and VR methods are still
Academic Editor: José
Gutiérrez-Maldonado
unclear. Although there is much research on brain correlates of executive impairments in schizophrenia
with traditional tools, it is uncertain if these results will be confirmed with the use of ecological and VR
Received: 10 May 2021 tools. In the diagnosis of executive dysfunction, it is important to use a variety of neuropsychological
Accepted: 18 June 2021 methods—especially those with confirmed ecological validity—to properly recognize the underlying
Published: 24 June 2021 characteristics of the observed deficits and to implement effective forms of therapy.

Publisher’s Note: MDPI stays neutral Keywords: virtual reality; schizophrenia; executive functions; cognitive functions; ecological validity;
with regard to jurisdictional claims in neuropsychology; psychopathological dimensions
published maps and institutional affil-
iations.

1. Introduction
Executive dysfunction, cognitive deficits, positive and negative symptoms, and disor-
Copyright: © 2021 by the authors. ganization are key psychopathological symptoms in schizophrenia; they are consequences
Licensee MDPI, Basel, Switzerland. of abnormalities in the structural and functional levels in the brain [1,2]. The cognitive
This article is an open access article deficits affect attention, verbal and episodic memory, and visuo-spatial processes [1,2]. It
distributed under the terms and
is worth noting that it has been claimed that executive dysfunction constitutes the most
conditions of the Creative Commons
specific set of neuropsychological symptoms in schizophrenia [3–5]. Although executive
Attribution (CC BY) license (https://
dysfunction is not included in the diagnostic criteria for schizophrenia in the Diagnostic
creativecommons.org/licenses/by/
and Statistical Manual of Mental Disorders-5th Edition (DSM-5) [6], the international group
4.0/).

J. Clin. Med. 2021, 10, 2782. https://doi.org/10.3390/jcm10132782 https://www.mdpi.com/journal/jcm

J. Clin. Med. 2021, 10, 2782 2 of 26

of experts who authored the manual define them as separate domains of cognitive impair-
ments and indicate that they are important aspects of schizophrenia. Investigating the
nature of executive dysfunction is very important, especially due to the fact that they affect
as many as 1% of the general population, regardless of country or cultural background,
causing significant impairment in the social, professional, and family life of patients and
leading to profound disability [7–9].
Currently, there is ongoing debate as to whether distinct types of schizophrenia can
be linked with specific executive disorders [10,11]. Numerous authors point out the need
to analyze particular subtypes of the disorder separately (e.g., those with predominantly
positive or negative symptoms [12]). Due to the heterogeneity of schizophrenia symptoms,
it seems valid to distinguish more homogeneous groups of patients within this one condi-
tion (such as negative and positive syndromes). Carpenter, Heinrichs, and Wagman [13]
propose the category of Deficit Schizophrenia (DS)—a subtype of the disorder with pre-
dominantly primary negative symptoms, which are stable over time. Negative symptoms
include poverty of speech, social withdrawal, limited speech content, restricted affect, and
apathy [14]. Numerous reports have demonstrated the validity of the diagnosis of DS,
due to, among other things, variations in genotype distribution [15], functional and struc-
tural differences in the brain [16], and cognitive and executive function differences [17,18].
However, research on the occurrence of executive dysfunction in DS has yielded inconsis-
tent results. Some studies suggest that, compared to non-deficit patients (NDS; patients
without deficit psychopathological symptoms), those with DS manifest difficulties with
regards to cognitive inhibition, problem-solving, and verbal and non-verbal cognitive
flexibility [19,20]; however, other studies have failed to confirm these results (for a detailed
overview see: [21]).
One of the most important challenges in contemporary neuropsychology is the use
of innovative technological methods to diagnose and treat schizophrenia patients [22].
Moreover, it is of utmost importance to develop tools that contribute to understanding
and predicting patients’ behavior in real-life situations [23]. There are several systematic
reviews [24,25], several selective reviews, and one meta-analysis [26] about assessment
and treatment using VR in different clinical populations. Other papers explore VR in the
schizophrenia spectrum in the broad context of different cognitive functions [27–29]. How-
ever, none of these seem to offer an extensive analysis of executive dysfunction measured
by VR in schizophrenia. To our knowledge, no previous papers detail the complex charac-
teristics of and compare traditional, ecological, and VR tools. Despite the well-established
links between executive impairments measured by traditional tools and psychopathologi-
cal dimensions e.g., [30], links between measurement by ecological and VR tools and the
aforementioned dimensions have been severely neglected in systematic reviews. This is
all the more surprising as knowing these relationships helps us understand the nature
of schizophrenia and to organize interdisciplinary diagnostic and support systems for
patients [31]. Moreover, systematic reviews lack information about the relationships of
white matter integrity or functional changes in the brain with executive functions measured
by ecological and VR tools. Successfully finding biological markers would enable faster
diagnosis, earlier implementation of effective treatment, and improved quality of life [32].
Given these limitations and the paucity of comprehensive analyses in the available review
papers, we formulated the following objectives. The primary aim of this review was to
characterize executive dysfunction in schizophrenia in conditions resembling everyday
situations by comparing traditional, ecological, and VR tools in terms of the ecological
dimensions of verisimilitude and veridicality. Our secondary goal was to describe the
neurobiological and psychopathological correlates of that dysfunction.

2. Nature of Executive Functions

Executive functions constitute an important subject of interest to researchers in var-
ious scientific disciplines. These processes are an important construct in modern neu-
ropsychology and they may be defined differently depending on the adopted theoretical
J. Clin. Med. 2021, 10, 2782 3 of 26

approach [33]. It is commonly accepted, however, that they mediate between cognition
(information processing systems) and action (i.e., the performance of activities) [34]. The
theoretical background of executive functions can be traced back to Luria’s research [35].
He distinguished three functional systems in the human brain: (a) an arousal and attention
unit (limbic and reticular activation system); (b) a unit that receives, stores, and analyzes in-
formation (posterior neocortex); and (c) a unit that plans, organizes, and regulates behavior
and cognition (frontal lobes). In his approach, regulation is defined as the ability to change
one’s actions during the execution of a particular task, which often takes place with the
involvement of linguistic processes [36]. The capacity to contrast the outcomes of a given
behavior with its initial purpose is defined as control. Finding a solution to a problem
involves analyzing initial conditions, developing a specific plan (strategy), implementing
the relevant operations, and comparing the result with the initial data [37].
On the other hand, Lezak, Howieson, and Loring [38] propose that executive functions
form a comprehensive set of processes or mental abilities used in the execution of purposeful
actions, enabling one to adapt to new situations. Lezak [38] clinically describes executive
functions in terms of four main domains: volition, purposive action, effective performance,
and planning. The first stage involves a conscious, future-oriented decision or intention
to carry out a deliberate action. The next stage, planning, happens once the goal has been
set: a sequence of steps necessary to solve the problem or achieve the goal is identified.
Purposive action is the process of turning intentions and plans into a specific behavioral
act; this requires the monitoring and correction of one’s course of action as well as the
modulation of the tempo and intensity of one’s own reactions. Another important element
of executive functions is working memory, introduced by Baddeley [39], which consists of a
central executive, phonological loop, and visual-spatial sketchpad. Working memory is also
considered an aspect of executive functions in the model proposed by Diamond [40].
Following these important formulations, researchers currently understand executive
functions as the mental abilities responsible for, inter alia: (a) planning and organization,
(b) anticipation and focus of attention, (c) initiation of activity, (d) self-monitoring and
impulse control, (e) working memory, (f) mental flexibility and the ability to make use of
feedback, and (g) choosing effective strategies for problem-solving [41]. These functions
also constitute an executive process that designates mental resources and monitors, inhibits,
and controls other mental processes and behavioral reactions, thereby reinforcing better
adaptation to the external environment [42]. Depending on the dynamics of a situation,
they facilitate the adjustment of behavior and tailoring of reactions [34]. Cognitive and
behavioral control constitute two especially important executive domains [43,44]. Studies
have shown a factorial structure of executive functions in healthy individuals by ana-
lyzing multiple measures of executive performance. For instance, Testa, Bennett, and
Ponsford [45] performed a factor analysis of 19 neuropsychological tests on 200 healthy
people and differentiated six largely independent factors: set-shifting and interference man-
agement, prospective working memory, task analysis, strategy generation and regulation,
response inhibition, and self-monitoring and set-maintenance. However, Miyake et al. [46]
distinguished only three central executive domains (shifting, inhibition, and updating).
The authors suggested the relative independence of these functions but noted that they
share certain common characteristics.
It is uncertain whether the same factor structure would be found using different
ecological and VR tools, and more research on this topic is needed. Additionally, the
factor structure of executive functions identified in healthy populations may differ from
that of clinical populations (e.g., for schizophrenia patients, see: [19]). Despite different
theoretical models of executive functions and repeated factor analyses, there are still
inconsistencies in their definitions and number of components; hence, “executive functions”
can be considered as an umbrella term indicating a “wide range of cognitive processes and
behavioral competencies” [33].
J. Clin. Med. 2021, 10, 2782 4 of 26

3. Brain Correlates of Executive Dysfunction in Schizophrenia

To improve the diagnosis and treatment of executive dysfunction in schizophrenia, it
is necessary to understand the complex relationship between executive functions and the
brain’s structure and activity [47]. Generally, the main cause of executive dysfunction is
believed to be abnormalities in different areas of the prefrontal cortex (e.g., dorsolateral,
ventrolateral, and anterior cingulate cortices) and their connections with other brain re-
gions, which are part of complex neural circuits [48]. On the basis of broad analysis of
evidence from structural and functional neuroimaging studies, Orellana and Slachevsky [4]
proposed dividing all neurocognitive theoretical models of schizophrenia into two cate-
gories. The first category includes neuroanatomical models that postulate that the executive
impairments are caused by dysregulation of specific brain circuits and regions: disruption
of the fronto-striato-thalamic system, disruption of the frontotemporal system, and dis-
ruption of the frontoparietal systems. The second category comprises cognitive models
that postulate that specific cognitive disorders are responsible for the symptomatology of
schizophrenia, such as Cohen’s model of updating internal representations of contextual
information and Frith’s model of disorder of consciousness or self-awareness that impairs
the ability to think with meta-representations.
One proposition that goes beyond the traditional approach is the functional neurosci-
entific paradigm in which links are drawn between particular brain regions and specific
functions. Recently, researchers have identified six brain neural networks involved in
cognitive functions, but according to the new paradigm, executive functions are performed
based on interactions between three large-scale brain networks, which do not have strictly
defined, localized frameworks [49,50]. This approach highlights the role of whole-brain
functional networks engaged in cognitive processes. These networks have been demon-
strated to include local neurons (located in areas of particular importance for specific
mental processes) that interact in synchrony with each other and at the same time stim-
ulate entire distal cell populations (in areas that are often remote from one another) [51].
The central executive network (CEN), consisting primarily of the dorsolateral prefrontal
cortex (DLPFC) and posterior parietal cortex, is activated in healthy subjects when per-
forming tasks involving executive functions associated with action-orientation and with a
primary focus towards the outside world [52]. In turn, the default mode network (DMN),
encompassing the ventromedial prefrontal cortex, posterior cingulate cortex, and pre-
cuneus, is deactivated during the performance of tasks that are associated with focus on
oneself [49,50]. Finally, the salience network (SN), involving activations in the anterior
insula and the dorsal anterior cingulate cortex, deals with the dynamic shifting between
the activity of the CEN and DMN networks, thus redirecting attention resources onto the
most important task at any particular moment [53]. One recent meta-analysis [54] of fMRI
research in schizophrenia has confirmed changes in the activity of structures involved in
these large-scale functional networks.
The disconnection hypothesis is a model of the etiopathogenesis of schizophrenia that
links it to the disturbance of the aforementioned neural networks [55]. Such impaired com-
munication is the result of, among other things, structural alterations during ontogenetic
development of the structures that constitute these functional networks [56,57]. Although
studies using functional neuroimaging have repeatedly suggested abnormalities in the
network of communication between the CEN, DMN, and SN in schizophrenia patients,
not much is currently known about the structural connections within these networks [58].
Executive disorders due to abnormalities in the structure and activity of the prefrontal
and thalamic networks are seen as the most characteristic neuropsychological symptoms
of schizophrenia [4,59]. Drawing on 41 functional magnetic resonance imaging studies,
Minzenberg et al. [60] demonstrated that executive deficits in schizophrenia are accompa-
nied by decreased activity in the left DLPFC, rostral and dorsal ACC, left thalamus, and
inferior/posterior cortical areas. Some reports indicate that, compared to healthy controls,
patients with schizophrenia have structural abnormalities within the CEN, including reduced
volume in the lateral PFC as well as in posterior areas of the parietal lobes [61] and reduced
J. Clin. Med. 2021, 10, 2782 5 of 26

integrity of the nerve fibers connecting these areas [62]. Other reports show that structural
abnormalities are present also in the cortical areas [63] and connections [64] of the DMN.
Finally, there are reports of structural abnormalities in the anterior cingulate and insula [65]
and the integrity of nerve fibers connecting the brain areas [66] that constitute the SN.
Reduced volume of the DLPFC and the ACC as well as reduced integrity of the cingu-
lum bundles (CB) in schizophrenia patients is linked to impaired ability to generate rules
and problems with abstract thinking [67–71]. Furthermore, reduced integrity of the frontal
portion of the CB [64,72] and the anterior limb of the internal capsule (ALIC) [62] has been
found to be linked to deficits in cognitive inhibition of dominant verbal responses. Other
reports have also shown that reduced integrity of the ALIC [73] and fornix [74] correlate
with impaired cognitive flexibility. However, some studies did not find a relationship
in patients with schizophrenia between executive functions and white matter integrity
in the superior longitudinal fasciculus, corpus callosum, anterior cingulum, and forceps
minor [75,76].

4. Old and New Paradigms for Assessing Executive Functions in Schizophrenia

Different paradigms for assessing executive functions have been discussed in the
field of neurocognitive psychology. Burgess et al. [77] describe three elements on which
neuropsychological assessment can focus: constructs, operations, and functions. Constructs
denote hypothetical cognitive resources whose existence is inferred from research findings
(e.g., cognitive inhibition). Operations are the individual component steps of cognition
that are not directly observable but can be inferred from a combination of task analysis and
changes in some dependent variable (i.e., an observable correlate of cognitive inhibition,
such as reaction time on the Stroop task). Functions, in contrast, are directly observable
behavioral output—the product of a series of operations. They are usually understood
in terms of a goal, instruction, or intention to act, and therefore it is usually clear what
constitutes success or failure, such as naming the color of words on the Stroop task. The
old paradigm, which aims to diagnose brain pathology, is the basis of most traditional
tools. Due to low process-behavior correspondence and non-material-specific processing,
the field of executive functions is especially heavily construct-oriented. The ecological
paradigm is, on the other hand, oriented toward functions, which is of great clinical
importance for diagnosing patients’ everyday cognitive abilities and disabilities (i.e., at
school or work). Research on the construct validity of ecological, function-oriented tools has
revealed correlations between traditional, construct-oriented tools and ecological ones, but
these associations turn out to be lower than ideal for establishing construct validity [78,79].
Despite the fact that tools developed in both paradigms measure executive functions,
they are understood in slightly different ways, which raises the question of whether they
actually measure the same construct.
The ecological paradigm highlights the importance of ecological validity in neu-
ropsychological testing, which is understood as “the degree to which test performance
corresponds to real-world performance” (p. 192, [80]). There is a need in neuropsychology
to develop research on the ecological validity of assessment and rehabilitation in the con-
text of clinical and forensic applications [81,82]. There are two prominent approaches to
defining crucial aspects of this type of validity. On the basis of the work of Kvavilashvili
and Ellis [83], Burgess et al. proposed the concepts of the representativeness of a task,
understood as “the extent to which a clinical test corresponds in form and context to a
situation encountered outside the laboratory” (p. 195, [77]), and the generalizability of test
results, understood as “the degree to which poor performance on the test will be predictive
of problems outside the laboratory” (p. 195, [77]). Franzen and Wilhelm [84] defined
ecological validity in terms of verisimilitude and veridicality. The concept of verisimilitude
refers to “the degree to which the cognitive demands of a test theoretically resemble the
cognitive demands in the everyday environment” (p. 182, [84]). Veridicality is defined
as “the degree to which existing tests are empirically related to measures of everyday
functioning” (p. 182, [84]). While veridicality can be measured by correlation with other
J. Clin. Med. 2021, 10, 2782 6 of 26

measures of functional outcome, including naturalistic observations, clinical scales, and

questionnaires, verisimilitude lacks empirical indicators, and a given tool can be considered
valid if it was constructed with verisimilitude in mind.
In this paper, we adopt the approach of Chaytor & Schmitter-Edgecombe [80] in
describing and classifying neurocognitive tools for measuring executive functions in
schizophrenia as it is more research-focused, whereas the approach of Burgess seems
more clinically oriented. We divided diagnostic tools into three basic categories: (a) tradi-
tional; (b) ecological; and (c) virtual reality (VR) methods, which can be understood as a
subclass of ecological tools, but due to their graphic realism and the actual activities the
participant undertakes in the environment, resulting in higher verisimilitude, we decided
to present them as a separate category. We chose ecological and VR tools on the basis of
their innovative character and clinical value in schizophrenia diagnosis; for traditional
tools, we selected the most used and researched ones. Figure 1 shows the tools used
to assess executive functions in schizophrenia in terms of the dimensions of verisimili-
tude and veridicality. Complementary Table 1 presents more detailed characteristics of
verisimilitude, evidence for verisimilitude, and diagnostic validity.

Figure 1. Tools for assessing executive functions in terms of their ecological validity level (verisimil-
itude and veridicality). Verisimilitude as a continuous dimension: stimuli and activities without
features of real-life—stimuli and activities with features of real-life. Veridicality as a continuous
dimension: lack of correlation of functional outcome and behavioral observation—correlation of
functional outcome and behavioral observation. CMPT = Computerized Meeting Preparation Task;
CST = Computerized Shopping Task; DS = Digit Span; DST = Digit Sequencing Test; EFPT = Ex-
ecutive Function Performance Test; LNST = Letter Number Sequencing Test; MET = Multiple Er-
rands Test; PDT = Plan-a-Day Test; SCWT = Stroop Color Word Test; SET = Six Elements Test;
SS = Spatial Span; TMT = Trail Making Test; TOL = Tower of London; VAP-S = Virtual Action
Plan—Supermarket; VCFMT = Virtual Cognitive Flexibility Measurement Task; VEP = Virtual Egyp-
tian Pyramids; VFT = Verbal Fluency Test; VMT = Virtual Maze Task; VRPMT = Virtual Reality
Prospective Memory Test; VSST = Virtual Supermarket Shopping Task; WCST = Wisconsin Card
Sorting Test; ZMT = Zoo Map Test.
J. Clin. Med. 2021, 10, 2782 7 of 26

Table 1. Characteristics of and evidence for diagnostic validity and two dimensions of ecological validity (verisimilitude and veridicality) of traditional, ecological, and VR tools that
measure executive functions in schizophrenia.

Ecological Validity
Diagnostic Validity
Type Tool Verisimilitude Veridicality
Examined Confirmed Developed Construct Technological Demands Examined Confirmed
Yes: significant difference Yes: significant correlation
between patients with Laboratory task: Computer version as between results on SCWT and
Traditional SCWT [85,86] Yes schizophrenia and healthy No paper-pencil or non-immersive Yes scales of community
controls [1,87,88]; several computerized task technology functioning [89–91]; several
meta-analyses meta-analyses
Yes: significant correlation
Yes: significant difference between results on TOL and
between patients with Laboratory task: Computer version as scales of community functioning,
Traditional TOL [85,92] Yes schizophrenia and healthy No paper-pencil or non-immersive Yes scales of social behavior in the
controls [93–95]; several computerized task technology milieu, scales of social
meta-analyses problem-solving, scales of social
skills [7]; one meta-analysis
Yes: significant difference Yes: significant correlation
between patients with Laboratory task: Computer version as between results on TMT and
Traditional TMT [85] Yes schizophrenia and healthy No paper-pencil or non-immersive Yes scales of community
controls [1,87,88]; several computerized task technology functioning [89,91]; two
meta-analyses meta-analyses
Yes: significant correlation
Yes: significant difference between results on WCST and
between patients with Laboratory task: Computer version as scales of community functioning,
Traditional WCST [85,96] Yes schizophrenia and healthy No paper-pencil or non-immersive Yes scales of social behavior in the
controls [1,2,87,88]; several computerized task technology milieu, scales of social
meta-analyses problem-solving, scales of social
skills [7]; one meta-analysis
Yes: significant difference Yes: significant correlation
between patients with Laboratory task: between results on VFT and
Traditional VFT [85,97] Yes schizophrenia and healthy No paper-pencil or Lack of computer version Yes scales of community
controls [1,87,88]; several computerized task functioning [7]; one
meta-analyses meta-analysis
J. Clin. Med. 2021, 10, 2782 8 of 26

Table 1. Cont.

Ecological Validity
Diagnostic Validity
Type Tool Verisimilitude Veridicality
Examined Confirmed Developed Construct Technological Demands Examined Confirmed
Yes: significant difference Yes: significant correlation
between patients with Laboratory task: Computer version as between results on DS and
Traditional DS [85,98] Yes schizophrenia and healthy No verbal or non-immersive Yes scales of community
controls [1,87,99,100]; several computerized task technology functioning [7,89–91]; several
meta-analyses meta-analyses
Yes: significant difference Yes: significant correlation
between patients with Laboratory task: between results on DST and
Traditional DST [101] schizophrenia and healthy No verbal or Lack of computer version Yes scales of community
controls [10]; one computerized task functioning [7]; one
meta-analysis meta-analysis
Yes: significant difference Yes: significant correlation
between patients with Laboratory task: Computer version as between results on LNST and
Traditional LNST [102] Yes schizophrenia and healthy No verbal or non-immersive Yes scales of community
controls [1]; one computerized task technology functioning [91]; one
meta-analysis meta-analysis
Yes: significant difference Yes: significant correlation
between patients with Laboratory task: Computer version as between results on SS and scales
Traditional SS [85,103] schizophrenia and healthy No manual or non-immersive Yes of community
controls [99,100]; two computerized task technology functioning [89–91]; several
meta-analyses meta-analyses
Independent living
as manual task:
Yes: significant difference
simple cooking, Computer version as
between patients with
Ecological EFPT [104,105] Yes Yes telephone use, non-immersive No No information
schizophrenia and healthy
medication technology
controls [104]; single study
management, and
bill payment
J. Clin. Med. 2021, 10, 2782 9 of 26

Table 1. Cont.

Ecological Validity
Diagnostic Validity
Type Tool Verisimilitude Veridicality
Examined Confirmed Developed Construct Technological Demands Examined Confirmed
Yes: significant correlation
between results on MET and
Yes: significant difference
two scales of real-world
between patients with Computer version as
Manual task: functioning [106]; single study
Ecological MET [106,107] Yes schizophrenia and healthy Yes non-immersive Yes
shopping center No: no significant correlation
controls [106,108]; two technology
between results in MET and two
studies
scales of real-world
functioning [108]; single study
Yes: significant correlation
between MSET and scale of
Yes: significant difference real-world functioning [111];
between patients with single study
Manual tasks: six
Ecological MSET [109] Yes schizophrenia and healthy Yes Lack of computer version Yes No: no significant correlation
realistic problems
controls [110]; one between results on MSET and
meta-analysis questionnaire ratings of
executive impairment [112];
single study
Yes: significant correlation
Yes: significant difference
between ZMT and questionnaire
between patients with
Manual task: map of ratings of executive impairment
Ecological ZMT [113,114] Yes schizophrenia and healthy Yes Lack of computer version Yes
zoo and scale of real-world
controls [110]; one
functioning [111,112];
meta-analysis
two studies
Virtual reality task: Computer non-immersive
Yes: significant correlation
Yes: significant difference supermarket with technology, scenario,
between results on VAP-S and
Virtual between patients with aisles displaying participants’ activities in
VAP-S [115] Yes Yes Yes results of naturalistic behavioral
reality schizophrenia and healthy items and static the environment, graphic
observations of real-life
controls [116]; single study virtual people representations of
tasks [117]; single study
(avatars) real items
J. Clin. Med. 2021, 10, 2782 10 of 26

Table 1. Cont.

Ecological Validity
Diagnostic Validity
Type Tool Verisimilitude Veridicality
Examined Confirmed Developed Construct Technological Demands Examined Confirmed
Computer non-immersive
Yes: significant difference Virtual reality task: technology, scenario,
Virtual between patients with city, station, bus and participants’ activities in
VCFMT [118] Yes Yes No No information
reality schizophrenia and healthy virtual people the environment, graphic
controls [118]; single study (avatars) representations of
real items
Computer non-immersive
Yes: significant difference Virtual reality task: technology, scenario, Yes: significant correlation
Virtual a between patients with office in a company participants’ activities in between results on CMPT and
CMPT [119] Yes Yes Yes
reality schizophrenia and healthy and virtual people the environment, graphic two scales of real-world
controls [119]; single study (avatars) representations of functioning [119]; single study
real items
Computer non-immersive
Yes: significant difference Virtual reality task: technology, scenario, No: no significant between
Virtual b between patients with office in a company participants’ activities in results on CMPT and two scales
CMPT [120] Yes Yes Yes
reality schizophrenia and healthy and virtual people the environment, graphic of real-world functioning [120];
controls [120]; single study (avatars) representations of single study
real items
Computer non-immersive
Yes: significant difference technology, scenario, Yes: significant correlation
Virtual reality task:
Virtual between patients with participants’ activities in between results on CST and
CST [121] Yes Yes grocery and virtual Yes
reality schizophrenia and healthy the environment, graphic scale of real-world
people (avatars)
controls [121]; single study representations of functioning [121]; single study
real items
Computer non-immersive
Yes: significant difference Yes: significant correlation
technology, scenario,
between patients with Virtual reality task: between results on PDT and
Virtual participants’ activities in
PDT [122] Yes schizophrenia and healthy Yes workplace Yes scale of real-world
reality the environment, graphic
controls [123,124]; environment functioning [123,124];
representations of
two studies two studies
real items
J. Clin. Med. 2021, 10, 2782 11 of 26

Table 1. Cont.

Ecological Validity
Diagnostic Validity
Type Tool Verisimilitude Veridicality
Examined Confirmed Developed Construct Technological Demands Examined Confirmed
Computer immersive
Yes: significant difference Virtual reality task: technology, scenario,
Virtual between patients with rooms with three participants’ activities in
VEP [125] Yes Yes No No information
reality schizophrenia and healthy doors in each and the environment, graphic
controls [125]; single study corridors representations of
real items
Yes: significant difference Computer immersive
Virtual reality task:
Virtual between patients with technology, scenario,
VMT [126] Yes Yes maze with walls and No No information
reality schizophrenia and healthy participants’ activities in
green grass
controls [126]; single study the environment
Virtual reality task: Computer non-immersive
Yes: significant difference shopping center with technology, scenario,
Virtual VRPMT [127, between patients with aisles displaying participants’ activities in
Yes Yes No No information
reality 128] schizophrenia and healthy items and static the environment, graphic
controls [128]; single study virtual people representations of
(avatars) real items
Virtual reality task: Computer non-immersive
Yes: significant difference supermarket with technology, scenario, No: no significant correlation
Virtual between patients with aisles displaying participants’ activities in between results on VSST and
VSST [129] Yes Yes Yes
reality schizophrenia and healthy items and static the environment, graphic scale of real-world
controls [129]; single study virtual people representations of functioning [129]
(avatars) real items
aCMPT without interruptions or unexpected outcomes and prospective memory instructions; b CMPT with interruptions or unexpected outcomes and prospective memory instructions. CMPT = Computerized
Meeting Preparation Task; CST = Computerized Shopping Task; DS = Digit Span; DST = Digit Sequencing Test; EFPT = Executive Function Performance Test; LNST = Letter Number Sequencing Test;
MET = Multiple Errands Test; PDT = Plan-a-Day Test; SCWT = Stroop Color Word Test; SET = Six Elements Test; SS = Spatial Span; TMT = Trail Making Test; TOL = Tower of London; VAP-S = Virtual Action
Plan—Supermarket; VCFMT = Virtual Cognitive Flexibility Measurement Task; VEP = Virtual Egyptian Pyramids; VFT = Verbal Fluency Test; VMT = Virtual Maze Task; VRPMT = Virtual-Reality Prospective
Memory Test; VSST = Virtual Supermarket Shopping Task; WCST = Wisconsin Card Sorting Test; ZMT = Zoo Map Test.
J. Clin. Med. 2021, 10, 2782 12 of 26

5. Assessment with Traditional Tools

Traditional pencil and paper tests are designed to measure executive functions in new
or non-standard situations. Regardless of the suggested theoretical approach, research
based on the use of standard methods indicates that patients with schizophrenia exhibit
disturbances in inhibition, monitoring, attention shifting, rule generation, abstract thinking,
planning, and working memory; these disturbances hamper the execution of goal-oriented
tasks (see: Table 2) [130,131]. Furthermore, due to the fact that executive symptoms in
patients with schizophrenia are often dissociated, meaning that the processes are subject
to differential impairment, patients are often diagnosed with “dysexecutive syndrome”, a
term often used to describe these different symptoms [132]. Such a diagnosis is based on
patients’ poor performance on traditional pencil and paper tests, such as the Stroop Color
Word Test (SCWT) [1,87,88], the Tower of London test (TOL) [93–95], the Trail Making Test
(TMT) [1,87,88], the Wisconsin Card Sorting Test (WCST) [1,2,87,88], the Verbal Fluency Test
(VFT) [1,87,88], the Digit Span test (DS) [1,87,99,100], the Digit Sequencing Test (DST) [10],
the Letter Number Sequencing Test (LNST) [1], and the Spatial Span test (SS) [99,100]. The
traditional neuropsychological tools used to assess executive functions in schizophrenia
are shown in Table 3.

Table 2. Components of executive dysfunction in schizophrenia [130,131].

Aspect of Executive Dysfunction Definition

Monitoring deficits Problems with the detection and resolution of conflict or detection and correction of errors.
Diminished ability to withhold automatic reactions or associations that are inappropriate to
Inhibition deficits
the currently executed task.
Attention shifting deficits Inability to flexibly transition between at least two processes or tasks.
Planning deficits Reduced ability to set sequences of steps that lead to the achievement of a particular goal.
Impaired formation of mental representations of rules concerning ways to solve problems in
Rule generation deficits
new situations.
Limitations in ability to break away from the current situational context and to transition to
Abstract thinking deficits
a different aspect that may not be directly related to the current state.
Storing and manipulating verbal or non-verbal information in a short period of time in
Working memory deficits
order to carry out a task.

Unfortunately, these tools typically have a similar structure and methodology to the
first methods used in the early twentieth century, (e.g., the Stanford-Binet Intelligence Scale)
and do not reflect patients’ difficulties in real life; in effect, these tests are characterized by
low levels of veridicality, confirmed by correlations on the level of 0.20–0.30 with functional
outcome scales (see: Table 1 and Figure 1) [85]. Additionally, these tools lack verisimili-
tude, as they were not originally designed for their current purpose—to assess executive
functions in patients with psychiatric disorders—because they were either constructed as
experimental investigations of basic cognitive systems in healthy persons (e.g., WCST) or
for testing frontal lobe functions in patients with anterior lesions (e.g., TOL). Therefore, it
might not be appropriate to draw conclusions concerning different aspects of executive
dysfunction in schizophrenia based on performance on such tests. Besides, since traditional
tools neglect the influence of emotional and social processes by diverging from the real-life
situations with which patients may have problems, it is impossible to assess the practical
effects of executive functions. Another limitation is that during standard psychological
testing, patients usually perform only one task, while, in real life, situations usually require
multitasking—carrying out a number of tasks simultaneously, which often compete with
each other at the same time [110,133]. Unlike in the laboratory, real-life problems are mostly
open-ended in nature, which is not reflected in the fixed goals and defined structure of
traditional tools [107]. Many patients with schizophrenia who achieve satisfactory results
on traditional tests can be impulsive, easily distracted, and have problems with using
J. Clin. Med. 2021, 10, 2782 13 of 26

feedback, which may be reflected in problems with real-life situations. These limitations
can be more adequately captured by more ecological tools [113]. Despite these limitations,
traditional tools are often used because of: familiarity, long tradition, availability, price,
short assessment time, and satisfactory diagnostic validity (see: Table 1).

Table 3. Description of traditional neuropsychological tools used to assess executive functions in schizophrenia.

Tool Description Components

Traditionally three independent tasks constitute the SCWT: quickly
Stroop Color Word identifying the names of colors in black ink; quickly identifying colors Working memory, verbal
Test (SCWT [85,86]) depicted as rectangles; and quickly identifying colored words printed attention, cognitive inhibition
in ink of an incongruent color. Manual and computer versions.
Participants are shown a wooden tower with three pins (small,
medium, large) and three balls (green, blue, red); from a
predetermined initial state, they are asked to move the balls, one by
Tower of London Planning, working memory,
one, to match a desired end position shown on a picture (12
(TOL [85,92]) visual attention
configuations in total), using the minimum necessary moves
(between 2 and 5); if the participant cannot finish or uses more moves,
they are asked to start over. Manual and computer versions.
This task is split into two parts. In part A, the participant connects 25
irregularly arranged circles containing numbers from 1 to 25; part B Non-verbal attention, working
Trail Making Test
requires alternation between circles arranged irregularly with memory, non-verbal cognitive
(TMT [85])
numbers (1 to 13) and letters (A to L) to form a continuous line. flexibility
Paper-and-pencil test.
WCST requires two identical decks of cards (each containing 64 cards)
Wisconsin Card
and four reference cards; a participant must place subsequent cards Working memory, switching,
Sorting Test
according to their shape, color, and number using the researcher’s perseveration, problem-solving
(WCST [85,96])
feedback. Manual and computer versions.
A letter (phonemic) or a category (semantic) task is used. Participants
Working memory, verbal
Verbal Fluency Test are asked to produce as many unique words as possible within 60 s
flexibility, word production,
(VFT [85,97]) from each category in the semantic task or words starting with a
verbal attention
given letter in the phonemic task.
Digit Span is a core subtest of the Wechsler Intelligence Scales. It
Verbal sustained attention,
consists of two parts: (a) Digits Forward characterized as a simple
storage and manipulation of
Digit Span (DS [85,98]) span test in which participants are asked to repeat a series of digits in
information in verbal working
a correct order, (b) Digits Backward that requires repeating digits in
memory
the reverse order of presentation.
Participants are asked to order series of numbers from lowest to Storage and manipulation of
Digit Sequencing Test
highest. Number of correct responses and the longest sequence information in non-verbal
(DST [101])
recalled correctly are recorded. working memory
Letter Number Participants are asked to sequence a random order of numbers and Storage and manipulation of
Sequencing Test letters (numbers in ascending order followed by letters in information in non-verbal
(LNST [102]) alphabetical order. working memory
Visual sustained attention,
Task consists of two parts: (a) Spatial Forward which requires
Spatial Span storage and manipulation of
participants to point the blocks in a given order, (b) Spatial Backward
(SS [85,103]) information in non-verbal
which requires pointing the blocks in the reverse order.
working memory

6. Assessment with Ecological Tools

Ecological methods are based on a new methodology that attempts to measure execu-
tive functions as they are used in real life situations. These methods are characterized by
a high level of representativeness of function, which means that they resemble situations
occurring outside the research laboratory, and verisimilitude, which indicates how well
they mirror the cognitive demands of authentic real-life challenges [80]. Patients with
schizophrenia have demonstrated limitations in their ability to solve open-ended problems,
difficulties performing simultaneous tasks, as well as deficits in planning, abstract think-
J. Clin. Med. 2021, 10, 2782 14 of 26

ing, and prospective memory (which, together, are necessary for completing operations
in conditions resembling true-to-life situations) in their poor performance on ecological
tests such as the Executive Function Performance Test (EFPT) [104], the Multiple Errands
Test (MET) [106,108], the Modified Six Elements Task (MSET) [110], and the Zoo Map
Test (ZMT) [110]. Descriptions of these ecological neuropsychological tools used to assess
executive functions in schizophrenia are given in Table 4.
Ecological tools have particularly good verisimilitude and veridicality (see: Table 1
and Figure 1); that is, performance in the task situation is predictive of performance in
situations outside it and the context and form of these tools corresponds to situations
encountered in real life—an important factor for clinical diagnosis. Ecological tools are
more useful for identifying executive dysfunction in patients with schizophrenia due to
the fact they measure executive skills in a more integrated way, through tasks that are
closer to real-life situations, instead of separated theoretical components (e.g., planning or
shifting) [107].

Table 4. Descriptions of the ecological neuropsychological tools used for measuring executive functions in schizophrenia.

Tool Description Components

Executive Function Completing four tasks related to self-maintenance and independent living: Working memory,
Performance Test simple cooking, using a telephone, managing medications, multitasking, planning,
(EFPT [104,105]) and paying bills. activities of daily living
This test measures everyday executive functioning with real-world
Strategy allocation,
Multiple Errands Test activities (e.g., purchasing specific items, arriving at a destination,
planning, working
(MET [106,107]) collecting and writing down information). The sum and type of errors
memory
(such as rule breaks and omissions) are registered.
This test consists of six tasks: two sets of arithmetic problems two sets of
Working memory, strategy
Modified Six Elements pictures to be named, and two dictation tasks; the participant attempts to
application, performance
Test (MSET [109]) solve them according to set rules within 10 min. Tasks of the same type
monitoring, planning
cannot be completed one after the other.
Zoo Map Test Participants plan a route for a visit to the zoo according to rules provided Working memory,
(ZMT [113,114]) (e.g., starting point, number of possible locations, using paths just once). planning, multitasking

7. Assessment with Virtual Reality

VR is an advanced interactive computer technology that generates a three-dimensional
environment [26]. It is characterized by a high level of immersion. Immersion is a mul-
tidimensional construct constituted by the objective characteristics of the virtual system,
perceptual responses to the virtual system, subjective responses to the narrative content,
and subjective responses to challenges in the virtual environment [134,135]. One’s subjec-
tive immersion in a virtual environment is a crucial feature of VR and is characterized by
two factors: place illusion, defined as subjective sense of presence in a virtual environment,
and plausibility illusion, namely the VR’s capacity to adequately react to the individual’s
behavior [25]. VR tools are often more complex than the traditional tests used to measure
executive functions. They are characterized by various non-linear goals, and thus are able
to measure multiple processes. Relatively lengthy immersion in a realistic environment
allows the accurate assessment of behavior in everyday situations [136].
There are three main types of VR assessments, which differ in terms of level of
immersion. The least immersive is a simple system that uses displays on monitors [28].
Moderately engaging methods involve the projection of images on a wall while participants
use shutter glasses to view the scene. The most immersive systems involve the use of
head-mounted displays [25]. The most engaging environments are used in the therapy
of various clinical groups, for example, schizophrenia and anxiety disorders. In scientific
research, simple and less immersive systems are most often used since they enable the
control of the basic characteristics necessary for strict experimental design [24].
J. Clin. Med. 2021, 10, 2782 15 of 26

Research based on VR methods has confirmed the presence of specific executive diffi-
culties in patients with schizophrenia, hindering their performance of complex everyday
activities, such as shopping, using public transport, or preparing a staff meeting in a busi-
ness office [24,25,137]. Although, like traditional tools, some VR tests were not developed
for patients with schizophrenia, evidence shows that they have good diagnostic validity
(see: Table 1). Using VR tools, it has been shown that schizophrenia patients have impaired
integration of visual and auditory stimuli. Moreover, Ku et al. [125] suggested that pa-
tients with schizophrenia have impaired multimodal integration and working memory
integration based on asking them to react to and interpret multimodal stimuli that had to
be memorized for a certain time period. Jeonghun Ku, Kim, and Kim [138] showed that
patients with schizophrenia cannot properly systematize or integrate the integrated stimuli
that occur in actual situations. A VR system was developed to administer multimodal
stimuli such as integrated visual and auditory stimuli. Patients were assessed and com-
pared in terms of their cognitive flexibility and working memory abilities by making them
experience and react to multimodal stimuli. Sorkin, Weinshall, Modai, and Peled [139]
examined sensory integration within schizophrenia patients’ working memory by means
of a virtual maze. Participants navigated through a series of rooms, repeatedly having to
choose from three doors. Each door had three characteristics (color, shape, and sound) and
only one combination of characteristics would open the door. Participants had to discover
the rule and implement it. When the task was over, each participant was given a perfor-
mance profile which included response time, error scores, strategy, and navigation ability.
Based on these profiles, a classification procedure was able to correctly identify 85% of the
schizophrenic patients. Furthermore, the patients did not exhibit any exceptional repetition
of responses regardless of the cessation of stimuli (traditionally known as “perseveration”),
as often happens in such patients. This only occurred when the task was not properly
explained to the participants.
Other specific executive difficulties in patients with schizophrenia have been demon-
strated in their poor performance on, inter alia, the Virtual-Action Planning Supermarket
(VAPS) [116], the Virtual Cognitive Flexibility Measurement Task (VCFMT) [118], the
Computerized Meeting Preparation Task (CMPT; see: Figure 2) [119,120], Computerized
Shopping Task (CST) [121], Plan-a-Day Test (PDT) [123,124], Virtual Egyptian Pyramids
(VEP) [125], Virtual Maze Task (VMT) [126], Virtual Reality Prospective Memory Test
(VRPMT) [128], and Virtual Supermarket Shopping Task (VSST; see: Figure 3) [129]. The
VR neuropsychological tools used to assess executive functions in schizophrenia are shown
in Table 5.

Table 5. Descriptions of virtual reality tools for measuring executive functions in schizophrenia.

Tool Description Components

This task emulates a medium size supermarket and the goal is to make
Virtual Action Cognitive planning,
purchases based on the list provided. Total distance, time, number of
Plan—Supermarket working memory,
products, correct and incorrect actions, and pauses are recorded. Computer
(VAP-S [115]) sustained attention
version.
The goal of this task is to get to a meeting place by bus; the participant is Decision making,
Virtual Cognitive
provided with a destination and recommended bus route. The participant cognitive flexibility,
Flexibility Measurement
can review a bus route map and choose one from a number of options prospective memory,
Task (VCFMT [118])
(wrong, slow, pre-informed, or quick). Computer version. working memory
The task consists of two parts: learning and meeting preparation. The
learning phase familiarizes the participant with the design of the task, which
involves progressive learning (total time and number of incorrect actions are
Computerized Meeting recorded). During the meeting preparation phase, the participant is asked to
Planning, multitasking,
Preparation Task organize a meeting according to the provided rules. The task requires, inter
working memory
(CMPT [119]) alia, placing guests, bringing office equipment, and preparing drinks. Total
time, number of times the instructions were checked, incorrectly
placed/missed objects, respect for rules, planning score, and initiation are
scored. Computer and real-life version.
J. Clin. Med. 2021, 10, 2782 16 of 26

Table 5. Cont.

Tool Description Components

This task consists of two parts. The learning phase aims to make the
participant familiar with the functions and actions available. For the main
shopping task, the participant is presented with a list of seven items Planning, multitasking,
Computerized Shopping belonging to distinct categories. Total time, correct items, intrusions, aisle organizing,
Task (CST [121]) redundancy, corrected errors, number of times in a non-relevant aisle, prospective memory,
number of times a non-relevant aisle was visualized, number of times the working memory
shopping list was checked, time spent checking the shopping list, and
checking the shopping cart are recorded. Computer version.
This task emulates a workplace environment. The goal of the task is to
Planning, organizing,
Plan-a-Day Test schedule a list of daily work-related activities based on a given list of tasks.
prospective memory,
(PDT [122]) Total time, planning ratio, and number of solved problems without
working memory
correction are recorded. Computer version.
Switching,
A participant walks through a series of rooms and corridors in a pyramid. perseveration,
Virtual Egyptian One of three doors needs to be chosen in each room. Doors are labeled with a problem-solving,
Pyramids (VEP [125]) shape, a color, and a sound. The task mechanics resemble the WCST. working memory,
Computer version. integration of
multimodal stimuli
The maze consists of six identical two-way intersections. Corridors lead
Spatial learning,
Virtual Maze Task either to another intersection or to one of seven cul-de-sacs. Only one dead
decision making,
(VMT [126]) end contains money (the goal). Five trials with a 5 min time limit are given to
working memory
reach the goal. Computer version.
An adaptation of the Virtual Reality Shopping Task [127]. It imitates a
Virtual Reality
shopping center with 20 stores. Obtaining items from a shopping list is used
Prospective Memory Prospective memory
as a distraction task. Execution of time-based and event-based prospective
Test (VRPMT [127,128])
memory tasks are recorded. Computer version.
Simulates shopping activity in a supermarket. Participants are asked to
Virtual Supermarket
memorize and collect items from a list. A shopping list cannot be consulted Planning, multitasking,
Shopping Task
and there is no time limit. Number of correct items, errors, trial time, and working memory
(VSST [129])
trial distance are scored. Computer version.

To summarize, VR methods involve measuring executive functions in virtual condi-

tions. They enable the simulation of physical space in the laboratory environment that
emulates the real world [27]. Moreover, such methods can allow participants to manipulate
three-dimensional objects and even interact with avatars representing other people; they
also give researchers the opportunity to make a more detailed analysis of the patient’s
behavior during the performance of a task [26]. Another important advantage of tools
based on this technology is the so-called sense of immersion, or the feeling of presence
in the virtual world [134]. VR assessment of executive functions tends to be similar to
those performed with other ecological methods in terms of, for example, multitasking,
open problems, and similarities to real situations. Executive functions measured in a
virtual environment (e.g., activities like shopping and managing finances) are linked to
the processes that are assessed by ecological tools [116,121,123]. Due to the fact that virtual
reality simulates natural environments and participants have a sense of immersion, it
makes possible the diagnosis of impairments of the executive functions that are used in
everyday activities. Although schizophrenia patients exhibit problems with cognitive and
behavioral inhibition, so far, most VR tools measure only planning, decision-making, and
multitasking. Future studies should investigate inhibition tasks in VR, using, for example,
the Stroop paradigm [78,140,141].
J. Clin. Med. 2021, 10, 2782 17 of 26

Figure 2. Sample pictures presenting the main room, person-avatar, and office material area from the Computerized Meeting
Preparation Task (CMPT) [119]. Reprinted with permission from [119]. Copyright 2014 Elsevier. Participant‘s first person
view: (A) in the main room where the meeting takes place, (B) in the kitchen, and (C) in the office material area.

In his recent work, Parsons [142] presented a list of potential risks of using VR in clini-
cal conditions, including: (a) simulator sickness, (b) high fidelity stimulus presentations,
experience intensification, and information overload, (c) depersonalization, derealization,
and dysfunctional re-entry into the real, (d) virtual environments with vulnerable pop-
ulations, and (e) therapeutic misconceptions. Additionally, VR tools tend to be costly,
especially highly immersive methods (e.g., head-mounted displays). Patients have to
learn how to use often complicated human-computer interfaces, which may be difficult
for individuals with severe perception and memory impairments. For older patients,
computer anxiety may also be part of the problem [143]. For the welfare of patients, it
is essential that technology support clinicians in their therapeutic efforts, not substitute
for them. The therapeutic relationship is exceedingly important for both diagnosis and
neuropsychological interventions.

Figure 3. Sample pictures presenting the participant’s first-person view while collecting items on the Virtual Supermarket
Shopping Task (VSST) [129]. The names of the products are visible when pointed at. Reprinted with permission from [129].
Copyright 2021 Czech Academy of Sciences. Participant‘s first person view: (A) while collecting an item from the shelf,
(B) while collecting fruits and vegetables, and (C) while collecting products from the fridge.

8. Relationship of Executive Dysfunction with Psychopathological Symptoms

Some studies have proposed that psychopathological symptoms could partially over-
lap with executive dysfunction in schizophrenia; for example, it has been found that
negative symptoms correlate with decreased abstract reasoning and impaired set shift-
ing [144], positive symptoms seem to only correlate with deficits in inhibition [145], and
disorganization symptoms correlate with difficulties in abstract thinking and impairments
in cognitive flexibility [146,147]. Some meta-analyses further confirmed the relationships of
both negative symptoms and disorganization with executive impairment measured using
traditional tests, with significant correlations being in the small-to-moderate range [30,148].
However, correlations for positive symptoms were nearly zero. The correlations between
various tests of executive functions were significantly different for disorganization and
negative symptoms. For example, there were stronger correlations between negative symp-
toms and the TMT, VFT, and SCWT than with the WCST and working memory tasks,
while there were stronger correlations between disorganization symptoms and the TMT,
J. Clin. Med. 2021, 10, 2782 18 of 26

SCWT, and WCST than with the VFT and working memory tasks. Patients who had stable
clinical profiles exhibited significantly stronger correlations with executive impairment
than did those with remitting or relapsing illnesses. Additionally, negative symptoms and
disorganization both correlated with general intellectual function (current IQ).
According to Thai et al. [110], only two studies have looked into the relationship
between psychopathological symptoms and executive dysfunction using ecological tests.
There were no significant correlations of subtests from the Behavioral Assessment of the
Dysexecutive Syndrome (BADS) battery with negative, positive, or disorganized symp-
toms [111,149]. In contrast, as assessed with Activities of Daily Living, negative symptoms
were linked to omissions during the choice of menu and greater delays between the first
and the last dishes, as well as the number of planning and repetition errors when cooking
the meal [150].
Only a few studies simultaneously focus on the link between psychopathological
symptoms and executive functions measured by VR tools in patients with schizophrenia.
For example, on the Computerized Shopping Task, Larøi et al. [121] showed that positive,
negative, and general symptoms were negatively correlated with the total times participants
consulted the list and the number of correct articles. Moreover, Josman et al. [116] showed
that negative symptoms correlated with all indices of the Virtual Action Plan—Supermarket
(VAP-S): trajectory duration, distance covered, purchases, correct actions, number of stops,
and total duration of stops. However, Laloyaux et al. [119] found no correlations between
psychopathological symptoms and different aspects of executive function as measured
by the Computerized Meeting Preparation Task (CMPT). A recent meta-analysis of the
Positive and Negative Syndrome Scale suggests five main psychopathological symptoms
in schizophrenia: negative symptoms, positive symptoms, disorganization, resistance,
and affect [151]. However, to the best of knowledge, this approach lacks data on the
relationship between psychopathological dimensions and results on VR tools that assess
executive functions in patients with schizophrenia.

9. Challenges and Future Directions

Despite the growing body of literature on executive function assessment and rehabil-
itation in schizophrenia, some unanswered questions still remain. One of the questions
concerns the relationship between executive deficits and structural abnormalities within
cortical areas of the brain and in the connections that create the three large-scale functional
networks. Most importantly, studies that have indicated such links have been conducted
using traditional methods to assess executive functions. There is little data on a possible
link between the structure and activity of the described neural networks and the executive
functions engaged by daily activities in patients with schizophrenia. Such data can be
collected only through ecological testing and VR methods. Future research should be
based on paradigms that allow the use of VR assessment of executive functioning with
methods of functional neuroimaging. This approach has been already used in research
on memory, learning, language production and comprehension, and social interaction
(with the use of functional brain neuroimaging, e.g., [126,152–155] and with event-related
potentials based on electroencephalography, e.g., [156,157]). Gainsford et al. [158] made the
interesting proposal of combining VR methods with non-invasive brain stimulation in the
social cognition domain, which would also be interesting to use to investigate executive
function in schizophrenia.
Technological advancements in the construction of tools have led to the emergence of
new questions and challenges. The questions concerning VR posed by Freeman [25] about
learning transfer from VR to the real world, the possibility of the application of complex
psychological treatment techniques to VR, and its benefits for patients in real life are still
valid. We consider the following research questions to currently be important in this field:
(a) How can the verisimilitude of neurocognitive tools be empirically verified? (b) What
are the relationships between the structure and activity of functional brain networks and
executive functions in real-life situations?
J. Clin. Med. 2021, 10, 2782 19 of 26

The literature analysis described in this paper reveals important issues that should be
considered in future research. One idea for measuring the verisimilitude of VR is to use a
panel of experts containing, in addition to clinicians, individuals who actually work in a
given environment (e.g., shop workers and regular customers in the case of supermarket
environments). As self-description questionnaires and clinical ratings do not provide a full
picture of a patient’s behavior outside the clinic, we suggest that veridicality should be as-
sessed in a more precise way, preferably by naturalistic observation in appropriately chosen
situations [117]. In addition to ecological validity, ecological tools, especially VR methods,
should be investigated in terms of the classical psychometric measures of different types
of reliability and validity, including diagnostic validity, in terms of sensitivity, specificity,
positive predictive power, and negative predictive power. In assessing test-retest reliability
for executive functions measures, it is important to control for the practice effect and to use
adequate statistical methods (e.g., the Reliable Change Index [159]). Similarly, Parsons [24]
highlighted the need for correspondence, representativeness, expedience, and relevance
in VR research. Future research should better control for social factors (e.g., interaction
with avatars) in VR research on executive functions. Concerning VR training of executive
functions, it is important to include follow-ups in the design of the research, in order to
understand the long term effects of such training. It is important to adopt an interdisci-
plinary approach, with more comprehensive behavior analysis, for research on executive
functions and rehabilitation of dysfunction thereof in schizophrenia patients. This should
be based on VR tools with complex real-life situation scripts combined with physiological
measurements (heart rate variability, galvanic skin response, and blood pressure) and brain
activity measurement carried out with different methods [158]. Augmented reality, which
allows the projection of items and avatars onto actual scenes, is another interesting new
direction for research.
Some clinical implications need to be discussed. Firstly, for both diagnosis and therapy,
it is important to find the right balance between maximizing patient benefit and avoiding
harm. Secondly, due to the lack of standardized VR tools for executive function assessment
in schizophrenia, which includes guidelines on development, norming, psychometric
validation, and administration, they should be used in combination with traditional tools
of proven psychometric value [26,142]. Similarly, due to the lack of empirically verified
and standardized therapeutic methods, VR tools should be used in combination with
traditional, empirically verified methods and cognitive-behavioral therapy. The American
Academy of Clinical Neuropsychology and the National Academy of Neuropsychology
advises that innovative computer diagnostic and therapeutic methods should always be
precisely described to provide comprehensive information on the product and thus to
allow clinicians to make informed decisions with patient welfare in mind [160].

10. Conclusions
Executive dysfunction in schizophrenia is an important clinical and social problem. Im-
pairments in monitoring, attention shifting, planning, inhibition, rule generation, abstract
thinking, and working memory, as well as the loss of skills required for the performance
of complex tasks, can negatively impact patients’ social and professional functioning as
well as their quality of life. When diagnosing executive dysfunction, an array of neu-
ropsychological methods must be used—not only traditional tools, but also those with
proven ecological validity (i.e., ecological and VR tools)—to accurately understand the
underlying nature of the deficits observed. However, due to the many potential risks of
using VR in clinical conditions and the lack of standardized VR tools for executive function
assessment in schizophrenia, they should be used in combination with traditional tools of
proven psychometric value. Future research should be based on paradigms that allow the
use of VR assessment of executive functioning with methods of functional neuroimaging.
Moreover, it would be interesting to investigate the influence of psychiatric medications on
executive functioning measured by VR techniques in schizophrenia.
J. Clin. Med. 2021, 10, 2782 20 of 26

Author Contributions: Conceptualization, methodology, writing—original draft preparation,

writing—review and editing, visualization, project administration, E.T.; writing—review and edit-
ing, M.M.; writing—review and editing, A.S. (Andrzej Sokołowski); writing—review and editing,
A.S. (Anna Starkowska); writing—review and editing, E.K.; writing—review and editing, M.K.;
writing—review and editing, Z.L.; writing—review and editing, J.P.; writing—review and editing
and supervision, L.S.; writing—review and editing and supervision, J.S.; writing—review and editing
and supervision, A.S.J. All authors have read and agreed to the published version of the manuscript.
Funding: This research was funded by the Faculty of Humanities at the University of Szczecin,
grant number 504-3000-240-940/2015 and 504-3000-240-940/2016, and the Pomeranian Medical
University in Szczecin, grant number FSN-337-06/2016 and FSN-246-05/2017, and the Polish Minister
of Science and Higher Education’s program named “Regional Initiative of Excellence” in 2019–
2022, grant number 002/RID/2018/2019 to the amount of 12 000 000 PLN and grant number
012/RID/2018/2019 to the amount of 11 985 347 PLN, and the National Science Centre in Poland,
grant number 2020/04/X/HS6/01920.
Institutional Review Board Statement: Not applicable.
Informed Consent Statement: Not applicable.
Data Availability Statement: Materials of the review reported here are available from the corre-
sponding author on reasonable request.
Conflicts of Interest: The authors declare no conflict of interest.

References
1. Dickinson, D.; Ramsey, M.E.; Gold, J.M. Overlooking the obvious: A meta-analytic comparison of digit symbol coding tasks and
other cognitive measures in schizophrenia. Arch. Gen. Psychiatry 2007, 64, 532–542. [CrossRef]
2. Fioravanti, M.; Bianchi, V.; Cinti, M.E. Cognitive deficits in schizophrenia: An updated metanalysis of the scientific evidence.
BMC Psychiatry 2012, 12, 64. [CrossRef] [PubMed]
3. Freedman, D.; Brown, A.S. The developmental course of executive functioning in schizophrenia. Int. J. Dev. Neurosci. 2011, 29,
237–243. [CrossRef] [PubMed]
4. Orellana, G.; Slachevsky, A. Executive functioning in schizophrenia. Front. Psychiatry 2013, 4, 35. [CrossRef] [PubMed]
5. Green, M.F.; Harvey, P.D. Cognition in schizophrenia: Past, present, and future. Schizophr. Res. Cogn. 2014, 1, 1–9. [CrossRef]
[PubMed]
6. American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders, 5th ed.; American Psychiatric Association
Publishing: Arlington, VA, USA, 2013; ISBN 9780890425596. [CrossRef]
7. Fett, A.K.; Viechtbauer, W.; Penn, D.L.; van Os, J.; Krabbendam, L. The relationship between neurocognition and social cognition
with functional outcomes in schizophrenia: A meta-analysis. Neurosci. Biobehav. Rev. 2011, 35, 573–588. [CrossRef]
8. Van der Werf, M.; Hanssen, M.; Köhler, S.; Verkaaik, M.; Verhey, F.R.; van Winkel, R.; van Os, J.; Allardyce, J. Systematic review
and collaborative recalculation of 133 693 incident cases of schizophrenia. Psychol. Med. 2014, 44, 9–16. [CrossRef] [PubMed]
9. Hay, S.I.; Abajobir, A.A.; Abate, K.H.; Abbafati, C.; Abbas, K.M.; Abd-Allah, F.; Abdulkader, R.S.; Abdulle, A.M.; Abebo, T.A.;
Abera, S.F.; et al. Global, regional, and national disability-adjusted life-years (DALYs) for 333 diseases and injuries and healthy
life expectancy (HALE) for 195 countries and territories, 1990–2016: A systematic analysis for the Global Burden of Disease Study
2016. Lancet 2017, 390, 1260–1344. [CrossRef]
10. Bora, E.; Akdede, B.B.; Alptekin, K. Neurocognitive impairment in deficit and non-deficit schizophrenia: A meta-analysis. Psychol.
Med. 2017, 47, 2401–2413. [CrossRef]
11. Kirkpatrick, B.; Galderisi, S. Deficit schizophrenia: An update. World Psychiatry 2008, 7, 143–147. [CrossRef] [PubMed]
12. Dollfus, S.; Lyne, J. Negative symptoms: History of the concept and their position in diagnosis of schizophrenia. Schizophr. Res.
2017, 186, 3–7. [CrossRef] [PubMed]
13. Carpenter, W.T.; Heinrichs, D.W.; Wagman, A.M. Deficit and nondeficit forms of schizophrenia: The concept. Am. J. Psychiatry
1988, 145, 578–583. [CrossRef] [PubMed]
14. Strauss, G.P.; Harrow, M.; Grossman, L.S.; Rosen, C. Periods of recovery in deficit syndrome schizophrenia: A 20-year multi–
follow-up longitudinal study. Schizophr. Bull. 2010, 36, 788–799. [CrossRef]
15. Pełka-Wysiecka, J.; Wroński, M.; Jasiewicz, A.; Grzywacz, A.; Tybura, P.; Kucharska-Mazur, J.; Bieńkowski, P.; Samochowiec, J.
BDNF rs 6265 polymorphism and COMT rs 4680 polymorphism in deficit schizophrenia in Polish sample. Pharmacol. Rep. 2013,
65, 1185–1193. [CrossRef]
16. Galderisi, S.; Quarantelli, M.; Volpe, U.; Mucci, A.; Cassano, G.B.; Invernizzi, G.; Rossi, A.; Vita, A.; Pini, S.; Cassano, P.; et al.
Patterns of structural MRI abnormalities in deficit and nondeficit schizophrenia. Schizophr. Bull. 2008, 34, 393–401. [CrossRef]
J. Clin. Med. 2021, 10, 2782 21 of 26

17. Cohen, A.S.; Saperstein, A.M.; Gold, J.M.; Kirkpatrick, B.; Carpenter, W.T., Jr.; Buchanan, R.W. Neuropsychology of the deficit
syndrome: New data and meta-analysis of findings to date. Schizophr. Bull. 2007, 33, 1201–1212. [CrossRef]
18. Yu, M.; Tang, X.; Wang, X.; Zhang, X.; Zhang, X.; Sha, W.; Yao, S.; Shu, N.; Zhang, X.; Zhang, Z. Neurocognitive impairments in
deficit and non-deficit schizophrenia and their relationships with symptom dimensions and other clinical variables. PLoS ONE
2015, 10, e0138357. [CrossRef] [PubMed]
19. Polgár, P.; Réthelyi, J.M.; Bálint, S.; Komlósi, S.; Czobor, P.; Bitter, I. Executive function in deficit schizophrenia: What do the
dimensions of the Wisconsin Card Sorting Test tell us? Schizophr. Res. 2010, 122, 85–93. [CrossRef] [PubMed]
20. Réthelyi, J.M.; Czobor, P.; Polgár, P.; Mersich, B.; Bálint, S.; Jekkel, É.; Magyar, K.; Mészáros, A.; Fábián, A.; Bitter, I. General and
domain-specific neurocognitive impairments in deficit and non-deficit schizophrenia. Eur. Arch. Psychiatry Clin. Neurosci. 2012,
262, 107–115. [CrossRef]
21. Tyburski, E.; Pełka-Wysiecka, J.; Mak, M.; Samochowiec, A.; Bieńkowski, P.; Samochowiec, J. Neuropsychological profile of
specific executive dysfunctions in patients with deficit and non-deficit schizophrenia. Front. Psychol. 2017, 8, 1459. [CrossRef]
[PubMed]
22. Vakil, E. Neuropsychological assessment: Principles, rationale, and challenges. J. Clin. Exp. Neuropsychol. 2012, 34, 135–150.
[CrossRef]
23. Ruff, R.M. A friendly critique of neuropsychology: Facing the challenges of our future. Arch. Clin. Neuropsychol. 2003, 18, 847–864.
[CrossRef] [PubMed]
24. Parsons, T.D.; Carlew, A.R.; Magtoto, J.; Stonecipher, K. The potential of function-led virtual environments for ecologically valid
measures of executive function in experimental and clinical neuropsychology. Neuropsychol. Rehabil. 2017, 27, 777–807. [CrossRef]
[PubMed]
25. Freeman, D.; Reeve, S.; Robinson, A.; Ehlers, A.; Clark, D.; Spanlang, B.; Slater, M. Virtual reality in the assessment, understanding,
and treatment of mental health disorders. Psychol. Med. 2017, 47, 2393–2400. [CrossRef]
26. Negut, , A.; Matu, S.A.; Sava, F.A.; David, D. Virtual reality measures in neuropsychological assessment: A meta-analytic review.
Clin. Neuropsychol. 2016, 30, 165–184. [CrossRef] [PubMed]
27. Macedo, M.; Marques, A.; Queirós, C. Virtual reality in assessment and treatment of schizophrenia: A systematic review. Braz. J.
Psychiatry 2015, 64, 70–81. [CrossRef]
28. Rus-Calafell, M.; Garety, P.; Sason, E.; Craig, T.J.; Valmaggia, L.R. Virtual reality in the assessment and treatment of psychosis: A
systematic review of its utility, acceptability and effectiveness. Psychol. Med. 2018, 48, 362–391. [CrossRef] [PubMed]
29. Kruk, D.; M˛etel, D.; Gaw˛eda, Ł.; Cechnicki, A. Implementation of virtual reality (VR) in diagnostics and therapy of nonaffective
psychoses. Psychiatr. Pol. 2020, 54, 951–975. [CrossRef] [PubMed]
30. Dibben, C.R.; Rice, C.; Laws, K.R.; McKenna, P.J. Is executive impairment associated with schizophrenic syndromes? A meta-
analysis. Psychol. Med. 2009, 39, 381–392. [CrossRef] [PubMed]
31. Harvey, P.D.; Koren, D.; Reichenberg, A.; Bowie, C.R. Negative symptoms and cognitive deficits: What is the nature of their
relationship? Schizophr. Bull. 2005, 32, 250–258. [CrossRef] [PubMed]
32. Galderisi, S.; DeLisi, L.E.; Borgwardt, S. Neuroimaging of Schizophrenia and Other Primary Psychotic Disorders; Springer: Cham,
Switzerland, 2019.
33. Chan, R.C.; Shum, D.; Toulopoulou, T.; Chen, E.Y. Assessment of executive functions: Review of instruments and identification of
critical issues. Arch. Clin. Neuropsychol. 2008, 23, 201–216. [CrossRef] [PubMed]
34. Jurado, M.B.; Rosselli, M. The elusive nature of executive functions: A review of our current understanding. Neuropsychol. Rev.
2007, 17, 213–233. [CrossRef] [PubMed]
35. Luria, A.R. Higher Cortical Functions in Man; Consultants Bureau: New York, NY, USA, 1980.
36. Luria, A.R. The Working Brain; Basic Books: New York, NY, USA, 1973.
37. Tupper, D.E. Introduction: Neuropsychological assessment apres Luria. Neuropsychol. Rev. 1999, 9, 57–61. [CrossRef] [PubMed]
38. Lezak, M.D.; Howieson, D.B.; Loring, D.W. Neuropsychological Assessment, 4th ed.; Oxford University Press: New York, NY, USA,
2004.
39. Baddeley, A. Working memory. Science 1992, 255, 556–559. [CrossRef] [PubMed]
40. Diamond, A. Executive functions. Annu. Rev. Psychol. 2013, 64, 135–168. [CrossRef]
41. Anderson, V.; Jacobs, R.; Anderson, P.J. Executive Functions and the Frontal Lobes: A Lifespan Perspective; Psychology Press: New
York, NY, USA, 2011.
42. Gioia, G.A.; Isquith, P.K.; Guy, S.C.; Kenworthy, L. Test review behavior rating inventory of executive function. Child Neuropsychol.
2000, 6, 235–238. [CrossRef]
43. Barkley, R.A. ADHD and the Nature of Self-Control; Guilford Press: New York, NY, USA, 1997.
44. Nigg, J.T. On inhibition/disinhibition in developmental psychopathology: Views from cognitive and personality psychology and
a working inhibition taxonomy. Psychol. Bull. 2000, 126, 220–246. [CrossRef]
45. Testa, R.; Bennett, P.; Ponsford, J. Factor analysis of nineteen executive function tests in a healthy adult population. Arch. Clin.
Neuropsychol. 2012, 27, 213–224. [CrossRef]
46. Miyake, A.; Friedman, N.P.; Emerson, M.J.; Witzki, A.H.; Howerter, A.; Wager, T.D. The unity and diversity of executive functions
and their contributions to complex “frontal lobe” tasks: A latent variable analysis. Cogn. Psychol. 2000, 41, 49–100. [CrossRef]
J. Clin. Med. 2021, 10, 2782 22 of 26

47. Millan, M.J.; Agid, Y.; Brüne, M.; Bullmore, E.T.; Carter, C.S.; Clayton, N.S.; Connor, R.; Davis, S.; Deakin, B.; DeRubeis, R.J.; et al.
Cognitive dysfunction in psychiatric disorders: Characteristics, causes and the quest for improved therapy. Nat. Rev. Drug Discov.
2012, 11, 141–168. [CrossRef]
48. Eisenberg, D.P.; Berman, K.F. Executive function, neural circuitry, and genetic mechanisms in schizophrenia. Neuropsychopharma-
cology 2010, 35, 258–277. [CrossRef] [PubMed]
49. Bressler, S.L.; Menon, V. Large-scale brain networks in cognition: Emerging methods and principles. Trends Cogn. Sci. 2010, 14,
277–290. [CrossRef]
50. Uddin, L.Q.; Yeo, B.T.; Spreng, R.N. Towards a universal taxonomy of macro-scale functional human brain networks. Brain Topogr.
2019, 32, 1–17. [CrossRef]
51. Bressler, S.L. Understanding cognition through large-scale cortical networks. Curr. Dir. Psychol. Sci. 2002, 11, 58–61. [CrossRef]
52. Uddin, L.Q.; Menon, V. The anterior insula in autism: Under-connected and under-examined. Neurosci. Biobehav. Rev. 2009, 33,
1198–1203. [CrossRef] [PubMed]
53. Jilka, S.R.; Scott, G.; Ham, T.; Pickering, A.; Bonnelle, V.; Braga, R.M.; Leech, R.; Sharp, D.J. Damage to the salience network and
interactions with the default mode network. J. Neurosci. 2014, 34, 10798–10807. [CrossRef]
54. O’Neill, A.; Mechelli, A.; Bhattacharyya, S. Dysconnectivity of large-scale functional networks in early psychosis: A meta-analysis.
Schizophr. Bull. 2019, 45, 579–590. [CrossRef]
55. Zhou, Y.; Fan, L.; Qiu, C.; Jiang, T. Prefrontal cortex and the dysconnectivity hypothesis of schizophrenia. Neurosci. Bull. 2015, 31,
207–219. [CrossRef] [PubMed]
56. Menon, V. Large-scale brain networks and psychopathology: A unifying triple network model. Trends Cogn. Sci. 2011, 15, 483–506.
[CrossRef]
57. Rapoport, J.L.; Giedd, J.N.; Gogtay, N. Neurodevelopmental model of schizophrenia: Update 2012. Mol. Psychiatry 2012, 17,
1228–1238. [CrossRef] [PubMed]
58. Miyata, J.; Sasamoto, A.; Mori, Y.; Isobe, M.; Sakai, Y.; Kochiyama, T.; Urayama, S.; Aso, T.; Murai, T.; Takahashi, H. 144. Structural
and Functional Connectivity and Delusional Cognitive Bias: A Multimodal Magnetic Resonance Imaging Study on Schizophrenia.
Schizophr. Bull. 2017, 43, S76. [CrossRef]
59. Giraldo-Chica, M.; Rogers, B.P.; Damon, S.M.; Landman, B.A.; Woodward, N.D. Prefrontal-thalamic anatomical connectivity and
executive cognitive function in schizophrenia. Biol. Psychiatry 2018, 83, 509–517. [CrossRef] [PubMed]
60. Minzenberg, M.J.; Laird, A.R.; Thelen, S.; Carter, C.S.; Glahn, D.C. Meta-analysis of 41 functional neuroimaging studies of
executive function in schizophrenia. Arch. Gen. Psychiatry 2009, 66, 811–822. [CrossRef] [PubMed]
61. Palaniyappan, L.; Mallikarjun, P.; Joseph, V.; White, T.P.; Liddle, P.F. Reality distortion is related to the structure of the salience
network in schizophrenia. Psychol. Med. 2011, 41, 1701–1708. [CrossRef] [PubMed]
62. Wagner, G.; De la Cruz, F.; Schachtzabel, C.; Güllmar, D.; Schultz, C.C.; Schlösser, R.G.; Bär, K.J.; Koch, K. Structural and functional
dysconnectivity of the fronto-thalamic system in schizophrenia: A DCM-DTI study. Cortex 2015, 66, 35–45. [CrossRef]
63. Salgado-Pineda, P.; Fakra, E.; Delaveau, P.; McKenna, P.J.; Pomarol-Clotet, E.; Blin, O. Correlated structural and functional brain
abnormalities in the default mode network in schizophrenia patients. Schizophr. Res. 2011, 125, 101–109. [CrossRef]
64. Drakesmith, M.; Caeyenberghs, K.; Dutt, A.; Zammit, S.; Evans, C.J.; Reichenberg, A.; Lewis, G.; David, A.S.; Jones, D.K.
Schizophrenia-like topological changes in the structural connectome of individuals with subclinical psychotic experiences. Hum.
Brain Mapp. 2015, 36, 2629–2643. [CrossRef]
65. Pu, W.; Li, L.; Zhang, H.; Ouyang, X.; Liu, H.; Zhao, J.; Li, L.; Xue, Z.; Xu, K.; Tang, H.; et al. Morphological and functional
abnormalities of salience network in the early-stage of paranoid schizophrenia. Schizophr. Res. 2012, 141, 15–21. [CrossRef]
66. Bracht, T.; Horn, H.; Strik, W.; Federspiel, A.; Razavi, N.; Stegmayer, K.; Wiest, R.; Dierks, T.; Müller, T.J.; Walther, S. White matter
pathway organization of the reward system is related to positive and negative symptoms in schizophrenia. Schizophr. Res. 2014,
153, 136–142. [CrossRef]
67. Rüsch, N.; Spoletini, I.; Wilke, M.; Bria, P.; Di Paola, M.; Di Iulio, F.; Martinotti, G.; Caltagirone, C.; Spalletta, G. Prefrontal–
thalamic–cerebellar gray matter networks and executive functioning in schizophrenia. Schizophr. Res. 2007, 93, 79–89. [CrossRef]
68. Nygård, M.; Løberg, E.M.; Craven, A.R.; Ersland, L.; Berle, J.Ø.; Kroken, R.A.; Johnsen, E.; Hugdahl, K. Dichotic listening,
executive functions and grey matter cortical volume in patients with schizophrenia and healthy controls. Scand. J. Psychol. 2013,
54, 443–450. [CrossRef]
69. Hatton, S.N.; Lagopoulos, J.; Hermens, D.F.; Hickie, I.B.; Scott, E.; Bennett, M.R. White matter tractography in early psychosis:
Clinical and neurocognitive associations. J. Psychiatry Neurosci. 2014, 39, 417–427. [CrossRef] [PubMed]
70. Molina, V.; Lubeiro, A.; Soto, O.; Rodriguez, M.; Álvarez, A.; Hernández, R.; de Luis-García, R. Alterations in prefrontal
connectivity in schizophrenia assessed using diffusion magnetic resonance imaging. Prog. Neuropsychopharmacol. Biol. Psychiatry
2017, 76, 107–115. [CrossRef]
71. Tyburski, E.; Mak, M.; Samochowiec, A.; Plichta, P.; Bielecki, M.; Rek-Owodzin, K.; Podwalski, P.; Rudkowski, K.; Pełka-Wysiecka,
J.; Kucharska-Mazur, J.; et al. The relationship between cingulum bundle integrity and different aspects of executive functions in
chronic schizophrenia. Prog. NeuroPsychopharmacol. Biol. Psychiatry 2020, 102, e109955. [CrossRef] [PubMed]
J. Clin. Med. 2021, 10, 2782 23 of 26

72. Takei, K.; Yamasue, H.; Abe, O.; Yamada, H.; Inoue, H.; Suga, M.; Muroi, M.; Sasaki, H.; Aoki, S.; Kasai, K. Structural disruption
of the dorsal cingulum bundle is associated with impaired Stroop performance in patients with schizophrenia. Schizophr. Res.
2009, 114, 119–127. [CrossRef]
73. Levitt, J.J.; Kubicki, M.; Nestor, P.G.; Ersner-Hershfield, H.; Westin, C.F.; Alvarado, J.L.; Kikinis, R.; Jolesz, F.A.; McCarley, R.W.;
Shenton, M.E. A diffusion tensor imaging study of the anterior limb of the internal capsule in schizophrenia. Psychiatry Res. 2010,
184, 143–150. [CrossRef] [PubMed]
74. Knöchel, C.; Schmied, C.; Linden, D.E.; Stäblein, M.; Prvulovic, D.; de Carvalho, L.D.; Harrison, O.; Junior, P.O.; Carvalho, A.F.;
Reif, A.; et al. White matter abnormalities in the fornix are linked to cognitive performance in SZ but not in BD disorder: An
exploratory analysis with DTI deterministic tractography. J. Affect. Disord. 2016, 201, 64–78. [CrossRef] [PubMed]
75. Prasad, K.M.; Upton, C.H.; Schirda, C.S.; Nimgaonkar, V.L.; Keshavan, M.S. White matter diffusivity and microarchitecture
among schizophrenia subjects and first-degree relatives. Schizophr. Res. 2015, 161, 70–75. [CrossRef] [PubMed]
76. Zhou, Y.; Liu, J.; Driesen, N.; Womer, F.; Chen, K.; Wang, Y.; Jiang, X.; Zhou, Q.; Bai, C.; Wang, D.; et al. White matter integrity in
genetic high-risk individuals and first-episode schizophrenia patients: Similarities and disassociations. BioMed Res. Int. 2017,
2017, 3107845. [CrossRef]
77. Burgess, P.W.; Alderman, N.; Forbes, C.; Costello, A.; Coates, L.A.; Dawson, D.R.; Anderson, N.D.; Gilbert, S.J.; Dumontheil, I.;
Channon, S. The case for the development and use of “ecologically valid” measures of executive function in experimental and
clinical neuropsychology. J. Int. Neuropsychol. Soc. 2006, 12, 194–209. [CrossRef] [PubMed]
78. Armstrong, C.M.; Reger, G.M.; Edwards, J.; Rizzo, A.A.; Courtney, C.G.; Parsons, T.D. Validity of the Virtual Reality Stroop Task
(VRST) in active duty military. J. Clin. Exp. Neuropsychol. 2013, 35, 113–123. [CrossRef] [PubMed]
79. Parsey, C.M.; Schmitter-Edgecombe, M. Applications of technology in neuropsychological assessment. Clin. Neuropsychol. 2013,
27, 1328–1361. [CrossRef] [PubMed]
80. Chaytor, N.; Schmitter-Edgecombe, M. The ecological validity of neuropsychological tests: A review of the literature on everyday
cognitive skills. Neuropsychol. Rev. 2003, 13, 181–197. [CrossRef]
81. Heinrichs, R.W. Current and emergent applications of neuropsychological assessment: Problems of validity and utility. Prof.
Psychol. Res. Pract. 1990, 21, 171–176. [CrossRef]
82. McCaffrey, R.J.; Williams, A.D.; Fisher, J.M.; Laing, L.C. The Practice of Forensic Neuropsychology: Meeting Challenges in the Courtroom;
Springer: Cham, Switzerland, 2004.
83. Kvavilashvili, L.; Ellis, J. Ecological validity and twenty years of real-life/laboratory controversy in memory research: A critical
(and historical) review. Hist. Philos. Psychol. 2004, 6, 59–80.
84. Franzen, M.D.; Wilhelm, K.L. Conceptual foundations of ecological validity in neuropsychology. In Ecological Validity of
Neuropsychological Testing; Sbordone, R.J., Long, C.J., Eds.; GR Press/St. Lucie Press: Delray Beach, FL, USA, 1996; pp. 91–112.
85. Strauss, E.; Sherman, E.; Spreen, O. A Compendium of Neuropsychological Tests: Administration, Norms, and Commentary; Oxford
University Press: Oxford, UK, 2006.
86. Stroop, J.R. Studies of interference in serial verbal reactions. J. Exp. Psychol. 1935, 18, 643–662. [CrossRef]
87. Fioravanti, M.; Carlone, O.; Vitale, B.; Cinti, M.E.; Clare, L. A meta-analysis of cognitive deficits in adults with a diagnosis of
schizophrenia. Neuropsychol. Rev. 2005, 15, 73–95. [CrossRef]
88. Knowles, E.E.; David, A.S.; Reichenberg, A. Processing speed deficits in schizophrenia: Reexamining the evidence. Am. J.
Psychiatry 2010, 167, 828–835. [CrossRef] [PubMed]
89. Green, M.F. What are the functional consequences of neurocognitive deficits in schizophrenia? Am. J. Psychiatry 1996, 153, 321–330.
[CrossRef] [PubMed]
90. Green, M.F.; Kern, R.S.; Braff, D.L.; Mintz, J. Neurocognitive deficits and functional outcome in schizophrenia: Are we measuring
the “right stuff”? Schizophr. Bull. 2000, 26, 119–136. [CrossRef]
91. Green, M.F.; Kern, R.S.; Heaton, R.K. Longitudinal studies of cognition and functional outcome in schizophrenia: Implications for
MATRICS. Schizophr. Res. 2004, 72, 41–51. [CrossRef] [PubMed]
92. Shallice, T. Specific impairments of planning. Philos. Trans. R. Soc. Lond. B 1982, 298, 199–209. [CrossRef]
93. Knapp, F.; Viechtbauer, W.; Leonhart, R.; Nitschke, K.; Kaller, C.P. Planning performance in schizophrenia patients. Psychol. Med.
2017, 47, 2002–2016. [CrossRef] [PubMed]
94. Rajji, T.K.; Ismail, Z.; Mulsant, B.H. Age at onset and cognition in schizophrenia: Meta-analysis. Br. J. Psychiatry 2009, 195,
286–293. [CrossRef]
95. Sullivan, J.R.; Riccio, C.A.; Castillo, C.L. Concurrent validity of the tower tasks as measures of executive function in adults: A
meta-analysis. Appl. Neuropsychol. 2009, 16, 62–75. [CrossRef]
96. Heaton, R.K.; Chelune, G.J.; Talley, J.L.; Kay, G.G.; Curtiss, G. Wisconsin Card Sort Test Manual: Revised and Expanded; Psychological
Assessment Resources: Odessa, TX, USA, 1993.
97. Tyburski, E.; Sokołowski, A.; Ch˛eć, M.; Pełka-Wysiecka, J.; Samochowiec, A. Neuropsychological characteristics of verbal and
non-verbal fluency in schizophrenia patients. Arch. Psychiatr. Nurs. 2015, 29, 33–38. [CrossRef]
98. Wechsler, D. Wechsler-Bellevue Intelligence Scale; The Psychological Corporation: New York, NY, USA, 1939.
99. Lee, J.; Park, S. Working memory impairments in schizophrenia: A meta-analysis. J. Abnorm. Psychol. 2005, 114, 599–611.
[CrossRef]
J. Clin. Med. 2021, 10, 2782 24 of 26

100. Forbes, N.F.; Carrick, L.A.; McIntosh, A.M.; Lawrie, S.M. Working memory in schizophrenia: A meta-analysis. Psychol. Med. 2009,
39, 889–905. [CrossRef] [PubMed]
101. Keefe, R.S.; Goldberg, T.E.; Harvey, P.D.; Gold, J.M.; Poe, M.P.; Coughenour, L. The Brief Assessment of Cognition in Schizophrenia:
Reliability, sensitivity, and comparison with a standard neurocognitive battery. Schizophr. Res. 2004, 68, 283–297. [CrossRef]
102. Nuechterlein, K.H.; Green, M.F.; Kern, R.S.; Baade, L.E.; Barch, D.M.; Cohen, J.D.; Essock, S.; Fenton, W.S.; Frese, F.J., 3rd; Gold,
J.M.; et al. The MATRICS Consensus Cognitive Battery, part 1: Test selection, reliability, and validity. Am. J. Psychiatry 2008, 165,
203–213. [CrossRef] [PubMed]
103. Wechsler, D. Manual for the Wechsler Memory Scale—Revised; The Psychological Corporation: New York, NY, USA, 1987.
104. Katz, N.; Tadmor, I.; Felzen, B.; Hartman-Maeir, A. Validity of the Executive Function Performance Test in individuals with
schizophrenia. OTJR 2007, 27, 44–51. [CrossRef]
105. Baum, C.M.; Morrison, T.; Hahn, M.; Edwards, D.F. Test Anual: Executive Function Performance Test; Washington University:
St. Louis, MO, USA, 2003.
106. Bulzacka, E.; Delourme, G.; Hutin, V.; Burban, N.; Méary, A.; Lajnef, M.; Leboyer, M.; Schürhoff, F. Clinical utility of the Multiple
Errands Test in schizophrenia: A preliminary assessment. Psychiatry Res. 2016, 240, 390–397. [CrossRef] [PubMed]
107. Shallice, T.I.; Burgess, P.W. Deficits in strategy application following frontal lobe damage in man. Brain 1991, 114, 727–741.
[CrossRef]
108. Caletti, E.; Paoli, R.A.; Fiorentini, A.; Cigliobianco, M.; Zugno, E.; Serati, M.; Orsenigo, G.; Grillo, P.; Zago, S.; Caldiroli, A.;
et al. Neuropsychology, social cognition and global functioning among bipolar, schizophrenic patients and healthy controls:
Preliminary data. Front. Hum. Neurosci. 2013, 7, 661. [CrossRef] [PubMed]
109. Burgess, P.W.; Alderman, N.; Evans, J.J.; Wilson, B.A.; Emslie, H.; Shallice, T. Modified Six Element Test. In Behavioural Assessment
of the Dysexecutive Syndrome; Wilson, B.A., Alderman, N., Burgess, P.W., Emslie, H., Evans, J.J., Eds.; Thames Valley Test Company:
Bury St Edmunds, UK, 1996; pp. 18–19.
110. Thai, M.L.; Andreassen, A.K.; Bliksted, V. A meta-analysis of executive dysfunction in patients with schizophrenia: Different
degree of impairment in the ecological subdomains of the Behavioural Assessment of the Dysexecutive Syndrome. Psychiatry Res.
2019, 272, 230–236. [CrossRef]
111. Vargas, M.L.; Sanz, J.C.; Marín, J.J. Behavioral assessment of the dysexecutive syndrome battery (BADS) in schizophrenia: A pilot
study in the Spanish population. Cogn. Behav. Neurol. 2009, 22, 95–100. [CrossRef]
112. Evans, J.J.; Chua, S.E.; McKenna, P.J.; Wilson, B.A. Assessment of the dysexecutive syndrome in schizophrenia. Psychol. Med.
1997, 27, 635–646. [CrossRef]
113. Wilson, B.A.; Evans, J.J.; Emslie, H.; Alderman, N.; Burgess, P. The development of an ecologically valid test for assessing patients
with a dysexecutive syndrome. Neuropsychol. Rehabil. 1998, 8, 213–228. [CrossRef]
114. Wilson, B.A.; Alderman, N.; Burgess, P.W.; Emslie, H.; Evans, J.J. Zoo map. In Behavioural Assessment of the Dysexecutive Syndrome;
Thames Valley Test Company: Bury St Edmunds, UK, 1996.
115. Klinger, E.; Chemin, I.; Lebreton, S.; Marié, R.M. A virtual supermarket to assess cognitive planning. Cyberpsychol. Behav. 2004, 7,
292–293. [CrossRef]
116. Josman, N.; Schenirderman, A.E.; Klinger, E.; Shevil, E. Using virtual reality to evaluate executive functioning among persons
with schizophrenia: A validity study. Schizophr. Res. 2009, 115, 270–277. [CrossRef]
117. Aubin, G.; Béliveau, M.F.; Klinger, E. An exploration of the ecological validity of the Virtual Action Planning–Supermarket
(VAP-S) with people with schizophrenia. Neuropsychol. Rehabil. 2018, 28, 689–708. [CrossRef]
118. Han, K.; Kim, I.Y.; Kim, J.J. Assessment of cognitive flexibility in real life using virtual reality: A comparison of healthy individuals
and schizophrenia patients. Comput. Biol. Med. 2012, 42, 841–847. [CrossRef]
119. Laloyaux, J.; Van der Linden, M.; Levaux, M.N.; Mourad, H.; Pirri, A.; Bertrand, H.; Domken, M.A.; Adam, S.; Larøi, F. Multitask-
ing capacities in persons diagnosed with schizophrenia: A preliminary examination of their neurocognitive underpinnings and
ability to predict real world functioning. Psychiatry Res. 2014, 217, 163–170. [CrossRef] [PubMed]
120. Laloyaux, J.; Van der Linden, M.; Nuechterlein, K.H.; Thonon, B.; Larøi, F. A direct examination of the cognitive underpinnings of
multitasking abilities: A first study examining schizophrenia. Psychiatry Res. 2018, 268, 288–296. [CrossRef] [PubMed]
121. Larøi, F.; Canlaire, J.; Mourad, H.; Van der Linden, M. Relations between a computerized shopping task and cognitive tests in
a group of persons diagnosed with schizophrenia compared with healthy controls. J. Int. Neuropsychol. Soc. 2010, 16, 180–189.
[CrossRef] [PubMed]
122. Funke, J.; Krüger, T. Plan-A-Day (Computer-Programm); Psychologisches Institut der Universität Bonn: Bonn, Germany, 1993.
123. Holt, D.V.; Wolf, J.; Funke, J.; Weisbrod, M.; Kaiser, S. Planning impairments in schizophrenia: Specificity, task independence and
functional relevance. Schizophr. Res. 2013, 149, 174–179. [CrossRef] [PubMed]
124. Holt, D.V.; Rodewald, K.; Rentrop, M.; Funke, J.; Weisbrod, M.; Kaiser, S. The Plan-a-Day approach to measuring planning ability
in patients with schizophrenia. J. Int. Neuropsychol. Soc. 2011, 17, 327–335. [CrossRef] [PubMed]
125. Ku, J.; Cho, W.; Kim, J.J.; Peled, A.; Wiederhold, B.K.; Wiederhold, M.D.; Kim, I.Y.; Lee, J.H.; Kim, S.I. A virtual environment for
investigating schizophrenic patients’ characteristics: Assessment of cognitive and navigation ability. CyberPsychol. Behav. 2003, 6,
397–404. [CrossRef] [PubMed]
J. Clin. Med. 2021, 10, 2782 25 of 26

126. Siemerkus, J.; Irle, E.; Schmidt-Samoa, C.; Dechent, P.; Weniger, G. Egocentric spatial learning in schizophrenia investigated with
functional magnetic resonance imaging. NeuroImage Clin. 2012, 1, 153–163. [CrossRef]
127. Canty, A.L.; Fleming, J.; Patterson, F.; Green, H.J.; Man, D.; Shum, D.H. Evaluation of a virtual reality prospective memory task
for use with individuals with severe traumatic brain injury. Neuropsychol. Rehabil. 2014, 24, 238–265. [CrossRef] [PubMed]
128. Man, D.W.; Ganesan, B.; Yip, C.C.; Lee, C.O.; Tsang, S.Y.; Yu, P.W.; Young, J.G.; Shum, D.H. Validation of the virtual-reality
prospective memory test (Hong Kong Chinese version) for individuals with first-episode schizophrenia. Neuropsychol. Rehabil.
2018, 28, 1197–1210. [CrossRef]
129. Plechatá, A.; Hejtmánek, L.; Fajnerová, I. Virtual supermarket shopping task for cognitive rehabilitation and assessment of
psychiatric patients: Validation in chronic schizophrenia. Cesk. Psychol. 2021, 65, 14–30. [CrossRef]
130. Kerns, J.G.; Nuechterlein, K.H.; Braver, T.S.; Barch, D.M. Executive functioning component mechanisms and schizophrenia. Biol.
Psychiatry 2008, 64, 26–33. [CrossRef] [PubMed]
131. Kim, J.; Glahn, D.C.; Nuechterlein, K.H.; Cannon, T.D. Maintenance and manipulation of information in schizophrenia: Further
evidence for impairment in the central executive component of working memory. Schizophr. Res. 2004, 68, 173–187. [CrossRef]
132. Reichenberg, A.A. The assessment of neuropsychological functioning in schizophrenia. Dialogues Clin. Neurosci. 2010, 12, e383.
133. Thornton, A.E.; Kristinsson, H.; DeFreitas, V.G.; Thornton, W.L. The ecological validity of everyday cognition in hospitalized
patients with serious mental illness. J. Clin. Exp. Neuropsychol. 2010, 32, 299–308. [CrossRef] [PubMed]
134. Nilsson, N.C.; Nordahl, R.; Serafin, S. Immersion revisited: A review of existing definitions of immersion and their relation to
different theories of presence. Hum. Technol. 2016, 12, 108–134. [CrossRef]
135. Slater, M. Immersion and the illusion of presence in virtual reality. Br. J. Psychol. 2018, 109, 431–433. [CrossRef]
136. Jansari, A.S.; Devlin, A.; Agnew, R.; Akesson, K.; Murphy, L.; Leadbetter, T. Ecological assessment of executive functions: A new
virtual reality paradigm. Brain Impair 2014, 15, 71–87. [CrossRef]
137. Van Bennekom, M.J.; de Koning, P.P.; Denys, D. Virtual reality objectifies the diagnosis of psychiatric disorders: A literature
review. Front. Psychiatry 2017, 8, 163. [CrossRef]
138. Jeonghun Ku, W.G.; Kim, J.H.; Kim, K.U. The development of a VR system for the cognitive & behavioral assessment of
schizophrenia. Stud. Health Technol. Inform. 2004, 98, 180–182. [CrossRef]
139. Sorkin, A.; Weinshall, D.; Modai, I.; Peled, A. Improving the accuracy of the diagnosis of schizophrenia by means of virtual reality.
Am. J. Psychiatry 2006, 163, 512–520. [CrossRef]
140. Henry, M.; Joyal, C.C.; Nolin, P. Development and initial assessment of a new paradigm for assessing cognitive and motor
inhibition: The bimodal virtual-reality Stroop. J. Neurosci. Methods 2012, 210, 125–131. [CrossRef]
141. Parsons, T.D.; Courtney, C.G.; Dawson, M.E. Virtual reality Stroop task for assessment of supervisory attentional processing. J.
Clin. Exp. Neuropsychol. 2013, 35, 812–826. [CrossRef]
142. Parsons, T.D. Ethical Challenges of Using Virtual Environments in the Assessment and Treatment of Psychopathological Disorders.
J. Clin. Med. 2021, 10, 378. [CrossRef] [PubMed]
143. Laguna, K.; Babcock, R.L. Computer anxiety in young and older adults: Implications for human-computer interactions in older
populations. Comput. Hum. Behav. 1997, 13, 317–326. [CrossRef]
144. Heydebrand, G.; Weiser, M.; Rabinowitz, J.; Hoff, A.L.; DeLisi, L.E.; Csernansky, J.G. Correlates of cognitive deficits in first
episode schizophrenia. Schizophr. Res. 2004, 68, 1–9. [CrossRef]
145. Waters, F.A.; Badcock, J.C.; Maybery, M.T.; Michie, P.T. Inhibition in schizophrenia: Association with auditory hallucinations.
Schizophr. Res. 2003, 62, 275–280. [CrossRef]
146. Good, K.P.; Rabinowitz, J.; Whitehorn, D.; Harvey, P.D.; DeSmedt, G.; Kopala, L.C. The relationship of neuropsychological test
performance with the PANSS in antipsychotic naıve, first-episode psychosis patients. Schizophr. Res. 2004, 68, 11–19. [CrossRef]
147. O’Leary, D.S.; Flaum, M.; Kesler, M.L.; Flashman, L.A.; Arndt, S.; Andreasen, N.C. Cognitive correlates of the negative,
disorganized, and psychotic symptom dimensions of schizophrenia. J. Neuropsychiatry Clin. Neurosci. 2000, 12, 4–15. [CrossRef]
148. Nieuwenstein, M.R.; Aleman, A.; de Haan, E.H. Relationship between symptom dimensions and neurocognitive functioning in
schizophrenia: A meta-analysis of WCST and CPT studies. J. Psychiatr. Res. 2001, 35, 119–125. [CrossRef]
149. Amann, B.L.; Gomar, J.J.; Ortiz Gil, J.; McKenna, P.J.; Sans-Sansa, B.; Sarró, S.; Moro, M.; Madre, M.; Landin-Romero, R.; Vieta,
E.; et al. Executive dysfunction and memory impairment in schizoaffective disorder: A comparison with bipolar disorder,
schizophrenia and healthy controls. Psychol. Med. 2012, 42, 2127–2135. [CrossRef]
150. Semkovska, M.; Bédard, M.A.; Godbout, L.; Limoge, F.; Stip, E. Assessment of executive dysfunction during activities of daily
living in schizophrenia. Schizophr. Res. 2004, 69, 289–300. [CrossRef] [PubMed]
151. Shafer, A.; Dazzi, F. Meta-analysis of the positive and Negative Syndrome Scale (PANSS) factor structure. J. Psychiatr. Res. 2019,
115, 113–120. [CrossRef]
152. Burgess, N.; Maguire, E.A.; Spiers, H.J.; O’Keefe, J. A temporoparietal and prefrontal network for retrieving the spatial context of
lifelike events. NeuroImage 2001, 14, 439–453. [CrossRef] [PubMed]
153. Folley, B.S.; Astur, R.; Jagannathan, K.; Calhoun, V.D.; Pearlson, G.D. Anomalous neural circuit function in schizophrenia during
a virtual Morris water task. NeuroImage 2010, 49, 3373–3384. [CrossRef] [PubMed]
J. Clin. Med. 2021, 10, 2782 26 of 26

154. Hawco, C.; Buchy, L.; Bodnar, M.; Izadi, S.; Dell’Elce, J.; Messina, K.; Joober, R.; Malla, A.; Lepage, M. Source retrieval is not
properly differentiated from object retrieval in early schizophrenia: An fMRI study using virtual reality. NeuroImage Clin. 2015, 7,
336–346. [CrossRef]
155. Salgado-Pineda, P.; Landin-Romero, R.; Portillo, F.; Bosque, C.; Pomes, A.; Spanlang, B.; Franquelo, J.C.; Teixido, C.; Sarró, S.;
Salvador, R.; et al. Examining hippocampal function in schizophrenia using a virtual reality spatial navigation task. Schizophr.
Res. 2016, 172, 86–93. [CrossRef]
156. Leong, V.; Schilbach, L. The promise of two-person neuroscience for developmental psychiatry: Using interaction-based
sociometrics to identify disorders of social interaction. Br. J. Psychiatry 2019, 215, 636–638. [CrossRef]
157. Meyer, L.; Lakatos, P.; He, Y. Language Dysfunction in Schizophrenia: Assessing Neural Tracking to Characterize the Underlying
Disorder (s)? Front. Neurosci. 2020, 15, 163. [CrossRef]
158. Gainsford, K.; Fitzgibbon, B.; Fitzgerald, P.B.; Hoy, K.E. Transforming treatments for schizophrenia: Virtual reality, brain
stimulation and social cognition. Psychiatry Res. 2020, 288, e112974. [CrossRef] [PubMed]
159. Duff, K. Evidence-based indicators of neuropsychological change in the individual patient: Relevant concepts and methods. Arch.
Clin. Neuropsychol. 2012, 27, 248–261. [CrossRef] [PubMed]
160. Bauer, R.M.; Iverson, G.L.; Cernich, A.N.; Binder, L.M.; Ruff, R.M.; Naugle, R.I. Computerized neuropsychological assess-
ment devices: Joint position paper of the American Academy of Clinical Neuropsychology and the National Academy of
Neuropsychology. Arch. Clin. Neuropsychol. 2012, 27, 362–373. [CrossRef] [PubMed]