International Psychogeriatrics (2020), 32:1, 141–149 © International Psychogeriatric Association 2019

doi:10.1017/S1041610219000504

Construct validity of the Visual Cognitive Assessment Test

(VCAT)—a cross-cultural language-neutral cognitive
screening tool
........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................

Audrey Low,1 Levinia Lim,1 Linda Lim,1 Benjamin Wong,1 Eveline Silva,1 Kok Pin Ng1 and
Nagaendran Kandiah1,2
1
Department of Neurology, National Neuroscience Institute, Singapore
2
Duke-NUS, Singapore

ABSTRACT

Background: The Visual Cognitive Assessment Test (VCAT) is a language-neutral cognitive screening tool
designed for use in culturally diverse populations without the need for translations or adaptations. While it has
been established to be language-neutral, the VCAT’s construct validity has not been investigated.
Methods: 471 participants were recruited, comprising 233 healthy comparisons, 117 mild cognitive impairment
(MCI), and 121 mild Alzheimer’s disease (AD) patients. VCAT and domain-specific neuropsychological tests
were administered in the same sitting. Construct validity was assessed by analyzing domain-specific associa-
tions between the VCAT and well-established cognitive assessments. Reliability (internal consistency) was
measured by Cronbach’s alpha. Diagnostic ability (area under the curve) and recommended cutoffs were
determined by receiver operating characteristic (ROC) analysis.
Results: The VCAT and its subdomains demonstrated good construct validity in terms of both convergent and
divergent validity and good internal consistency (α = .74). ROC analysis found that the VCAT was on par with
the Mini Mental State Examination (MMSE) and Montreal Cognitive Assessment (MoCA) at distinguishing
between healthy comparisons, MCI, and mild AD. Consistent with previous studies, VCAT scores were not
affected by language of administration or ethnicity in our cohort. Findings suggest the following cutoffs:
Dementia 0–19, MCI 20–24, Normal 25–30.
Conclusion: This study established the construct validity of the VCAT, which is vital to ensure its subdomains
effectively measure the cognitive processes they were designed to. The VCAT is capable of detecting early
cognitive impairments and allows for meaningful cross-cultural comparisons, especially useful for international
collaborations and clinical trials, and for clinical use in diverse multiethnic populations.

Key words: dementia, mild cognitive impairment, Alzheimer disease, neuropsychological assessment, reliability, diagnostic value

Introduction diagnostic tools are available for use in these lin-

guistically and culturally diverse populations.
Worldwide, the number of people with dementia is However, most cognitive screening tools are lack-
projected to double every 20 years (Prince et al., ing in diagnostic accuracy when used in non-
2013). Notably, much of this increase will be a result Caucasian populations (Boone et al., 2007; Brickman
of the increasing prevalence of dementia in low- and et al., 2006; Loewenstein et al., 1994; Manly, 2005).
middle-income countries, many of which are in This may stem from the fact that the majority of
Asia, Latin America, North Africa, and the Middle
today’s cognitive screening tools have been developed
East. Thus, it is important to ensure that accurate
for English-speaking populations. Often, test devel-
opers fail to consider linguistic and cultural differ-
ences and their impact on construct validity and test
Correspondence should be addressed to: Nagaendran Kandiah, Senior Consultant
transferability when used internationally. As a result,
Neurologist, Department of Neurology, National Neuroscience Institute, 11 non-Westerners are often assessed on tasks that do
Jalan Tan Tock Seng, Singapore 308433. Phone: +65 6357 7199; Fax: +65 not translate meaningfully in their culture.
6357 7137. Email: [email protected]. Received 31 Dec 2018;
revised version received 13 Feb 2019; accepted 28 Mar 2019. First published Individuals tested with stimuli uncommon in
online 21 May 2019. their culture may be unfairly penalized, and hence

https://doi.org/10.1017/S1041610219000504 Published online by Cambridge University Press

 142 A. Low et al.

wrongly diagnosed, due to unfamiliarity with the complete (Kandiah et al., 2015). The VCAT is
stimuli. A review by Manly and Espino found that scored from 0 to 30, whereby a higher score denotes
the ethnic differences on neuropsychological tests higher cognitive functioning. Informed by common
persist even when groups are matched on sex, age, diagnostic criteria used to diagnose mild cognitive
education, and socioeconomic status (Manly and impairment (MCI) and Alzheimer’s disease (AD),
Espino, 2004). Furthermore, Gibbons and collea- the VCAT was designed to measure five specific
gues observed differences in the Mini Mental State cognitive domains: episodic memory, attention/
Examination (MMSE) item scores even when working memory (WM), executive function, visuo-
comparing two English-speaking cohorts (United spatial ability, and language. The test is comprised of
Kingdom and United States of America), attributing three episodic memory items (13 points), one atten-
the findings to translation artifacts stemming from tion item (3 points), three executive functioning
cultural differences (Gibbons et al., 2002). items (6 points), two visuospatial items (3 points),
Even when word lists are replaced with culturally and two language items (5 points). The VCAT was
appropriate words, they may differ from the original designed to be weighted more heavily toward the
in terms of length, imagery, phonological complex- episodic memory domain, as memory impairments,
ity, and ease of semantic clustering with other words in particular, are predictive of AD conversion, mak-
in the list. Similar difficulties exist for digit-based ing the VCAT especially useful in the early diagnosis
tasks. For example, it has been well-established that of AD. Emphasis was also placed on the executive
Mandarin speakers perform better than English functions domain to better detect vascular demen-
speakers on digit-based working memory tasks, tia. Instructions are communicated in a language
such as the Wechsler Adult Intelligence Scale – that both the test administrator and the participant
Fourth Edition (WAIS-IV) Digit Span, even when are comfortable conversing in.
matched on race—many have attributed this to the While the VCAT has been established to be
monosyllabic feature of the Mandarin language and language-neutral (Lim et al., 2018), it has not
the shorter length of articulation of digits (Hoosain, been assessed in terms of construct validity. As
1982; Stigler et al., 1986; Ting et al., 2013). Another such, this present study aims to: (1) assess the
study in Ireland administered both the English and construct validity of the VCAT and its subdomains
Irish version of the MMSE to bilingual participants by measuring patterns of association between the
and found significant individual and group differences VCAT domains and other established measures of
between test scores (Ní Chaoimh et al., 2015). These the same cognitive domains; and (2) evaluate the
linguistic and cultural issues make it difficult to not diagnostic value of the test in discriminating healthy
only compare individuals across countries but also from cognitively impaired participants, and MCI
within countries of diverse multiethnic populations. from mild AD participants.
As such, there is an urgent need for standardized
cognitive screening tools that are transferable across
countries, language, and cultures without any need Methods
for translations or adaptations (Franzen et al., 2017;
Romero et al., 2009; Shah et al., 2008; Sperber, 2004; Participants
Teng, 2002). Of the 471 participants in the study sample, 233
were healthy comparisons (HCs), 117 were patients
with MCI, and 121 were mild AD. HCs were
Description of VCAT
recruited from the community, while MCI and
The Visual Cognitive Assessment Test (VCAT) was mild AD participants were recruited from a special-
developed as a solution to the above-mentioned ist outpatient memory clinic, between the period of
problems faced by existing cognitive screening tools October 2013 and November 2017 (National Neu-
(Kandiah et al., 2015). To address the problems roscience Institute, Singapore).
often accompanying translation, the VCAT was Classifications of MCI and mild AD were based
designed to be a language-neutral cognitive screen- on clinical diagnosis by cognitive neurologists
ing tool, negating the need for translations. The test based on the National Institute on Aging and
was also designed to be visual-based so as to reduce Alzheimer’s Association (NIA-AA) criteria for MCI
language demands. In a previous multinational and NIA-AA criteria for AD (Albert et al., 2011;
study of four Southeast Asian countries, Montreal McKhann et al., 2011). MCI patients were required
Cognitive Assessment (MoCA) scores differed by to have a Clinical Dementia Rating (CDR) score of
language of administration, while performance on 0.5 and mild AD patients were required to have a
the VCAT was not (Lim et al., 2018). CDR score of 1 (Morris, 1993). All classifications
The VCAT is an 11-item visual-based cognitive were corroborated by comprehensive neuropsycho-
screening tool, which takes 15.7±7.3 minutes to logical assessment and supported by structural

https://doi.org/10.1017/S1041610219000504 Published online by Cambridge University Press

 Construct validity of VCAT 143

magnetic resonance imaging (MRI) scans where Statistical analysis

available. Healthy older adults were recruited Statistical analysis was performed using SPSS
from the community and were required to have (Version 20.0, SPSS, Inc., Chicago, IL, U.S.A.).
no cognitive complaints, and a CDR score of 0. Continuous test variables were assessed for normal-
This study was approved by the Singhealth institu- ity using the Shapiro–Wilk test. ANOVA was used to
tional ethics review board, and informed consent perform between-group comparisons on normally
was obtained from all patients prior to data distributed continuous data, while the Kruskal–
collection. Wallis test was used to analyze non-normal contin-
uous data—age, education, and all VCAT domain
Neuropsychological assessment and VCAT individual item scores were not normally
To assess its construct validity, the VCAT was distributed. Categorical data were analyzed using χ2
administered alongside a battery of other neuropsy- tests of independence. Multiple linear regression
chological assessments (NPA), comprised of well- models were used to adjust for common confoun-
established tests to measure each of the five cognitive ders, such as gender, race, age, and years of educa-
domains included in the VCAT. Cognitive tests tion. To avoid Type 1 errors arising from multiple
were administered by trained psychologists in a comparisons, the Holm–Bonferroni sequential cor-
quiet room free of distraction. Global cognition was rection was applied. To evaluate construct validity,
assessed using the MMSE and the Hong Kong correlation analysis was conducted to assess the
adaptation of the MoCA. Episodic memory function strength of associations between VCAT cognitive
was measured using the 10-word Immediate and domains and other established cognitive measures.
Delayed Recall task and the Word Recognition task To assess reliability, we examined the internal con-
of the Alzheimer’s Disease Assessment Scale- sistency, as measured by the Cronbach’s alpha, and
Cognitive Subscale (ADAS-Cog), the Logical the inter-item correlation matrix. Receiver operating
Memory Immediate and Delayed Recall task of characteristic (ROC) analysis was used to determine
the Wechsler Memory Scale–Fourth Edition the best VCAT cutoffs to differentiate: (1) HCs from
(WMS-IV), and the Rey–Osterrieth Complex Fig- patients with cognitive impairment (MCI+AD), and
ure Task (RCFT) Immediate Recall, Delayed (2) MCI from mild AD. Diagnostic performance
Recall, and Recognition task. For further analysis, was also assessed using the area under the curve
the memory domain was further broken down into (AUC), positive (PPV) and negative predictive va-
Visual-Auditory Memory (composite of ADAS-Cog lues (NPV), as well as positive (PLR) and negative
tasks), Auditory Memory (composite of WMS-IV (NLR) likelihood ratios. Optimal cutoff scores were
Logical Memory tasks), and Visual Memory (com- defined as maximal sensitivity and specificity, mea-
posite of RCFT tasks). Attention was assessed using sured by Youden’s Index. Significance for all signif-
the Color Trails Test 1 and the WAIS-IV Digit Span icance tests performed was set at a two-tailed
Forward. Executive function was measured using probability value of 0.05.
the Color Trails Test 2, the Frontal Assessment
Battery, WAIS-IV Digit Span Backward, and the
Executive Clock-Drawing Task (CLOX1). Visuo- Results
spatial abilities were assessed using the WAIS-IV
Block Design subtest, RCFT Copy, and the Incom- Characteristics of study sample
plete Letters and Dot Counting tasks of the Visual The total sample was comprised of 471 participants.
Object and Space Perception Battery. Language Between-group comparisons found significant gender
was evaluated using the Naming subtest of the differences between the three groups (Table 1), with a
ADAS-Cog, the Boston Naming Test, a test of higher proportion of males in the MCI group (60.7%),
phonemic fluency, and a test of semantic fluency. compared to the HC group (30.0%) and the mild AD
Only a subset of participants were assessed on Visual group (37.2%). In terms of age, the mild AD group
Memory (n = 78) and visuospatial abilities (n = 102). was significantly older (mean = 70.25, SD = 9.22)
However, given the visual-based nature of the VCAT, compared to HC (mean = 61.47, SD = 7.19) and
we felt it was pertinent to retain these tests in assessing MCI (mean = 63.28, SD = 9.05). Participants in the
the VCAT’s construct validity. To compare the mild AD group also tended to have fewer years of
VCAT subdomains with the cognitive domains of formal education (mean = 8.39, SD = 5.09) com-
other cognitive tests, composite z-scores of each pared to HC (mean = 12.36, SD = 3.76) and MCI
domain were computed. Individual raw scores of (mean = 11.48, SD = 4.54). The three diagnostic
the various tests were first transformed into z-scores groups performed significantly differently on total
based on local norms, which were then averaged to VCAT score, all its subdomains, and most of its
compute a composite domain z-score. individual items, even after controlling for race, age,

https://doi.org/10.1017/S1041610219000504 Published online by Cambridge University Press

 144 A. Low et al.

Table 1. Participant characteristics

WHOLE SAMPLE HC MCI M I L D AD
( N = 471) ( N = 233) ( N = 117) ( N = 121) P
...................................................................................................................................................................................................................................................................................................................................

Gender1 Male % 39.5 30.0 60.7 37.2 <0.001

Age2 Mean in years (SD) 64.00 (8.92) 61.47 (7.19) 63.28 (9.05) 70.25 (9.22) <0.001
Race1 — — — — 0.186
Chinese % 92.5 93.9 90.6 91.7 —
Malay % 2.6 1.7 3.4 3.3 —
Indian % 2.6 1.7 4.3 2.5 —
Others % 2.4 2.6 1.7 2.5 —
Education2 Mean in years (SD) 11.01 (4.68) 12.36 (3.76) 11.48 (4.54) 8.39 (5.09) <.001
MMSE 27.00 (2.59) 28.26 (1.71) 27.04 (2.21) 24.53 (2.60) <.001
MoCA 25.50 (3.74) 27.64 (2.03) 25.45 (2.89) 21.37 (3.54) <.001
Note: For continuous variables, that is, age and education, the Shapiro–Wilk test was performed to test for normality. Both variables were
demonstrated to have non-normal distributions (p < 0.05) and were therefore analyzed using the Kruskal–Wallis test.
χ test.
1 2
2
Kruskal–Wallis test.

Table 2. VCAT subdomain and individual item scores

P ADJUSTED FOR RACE,
RANGE HC MCI MILD AD P AGE, EDUCATION
...................................................................................................................................................................................................................................................................................................................................

VCAT total 0–30 26.16 (3.34) 22.55 (4.93) 17.21 (4.78) <0.001 <0.001**
VCAT subdomains
Memory 0–13 11.74 (1.43) 10.28 (2.34) 7.50 (3.11) <0.001 <0.001**
Attention/WM 0–3 2.29 (1.16) 1.88 (1.36) 0.93 (1.30) <0.001 <0.001**
Executive function 0–6 5.00 (1.28) 4.29 (1.68) 3.28 (1.59) <0.001 <0.001**
Visuospatial 0–3 2.70 (0.50) 2.55 (0.58) 2.18 (0.64) <0.001 <0.001**
Language 0–5 4.62 (0.63) 4.15 (0.89) 3.55 (1.03) <0.001 <0.001**
VCAT items
Memory–Shapes 0–2 1.28 (0.91) 0.86 (0.87) 0.25 (0.58) <0.001 <0.001**
Memory–Objects 0–8 7.50 (0.84) 6.27 (1.87) 4.41 (2.59) <0.001 <0.001
Memory–Scene 0–3 2.98 (0.23) 2.97 (0.16) 2.62 (0.67) <0.001 <0.001**
Attention–Shape cancellation 0–3 2.30 (1.14) 1.62 (1.38) 0.77 (1.21) <0.001 <0.001**
Executive function–Gears 0–3 2.21 (1.06) 1.65 (1.23) 1.65 (1.10) <0.001 0.075
Executive function–Pattern 0–2 1.77 (0.54) 1.35 (0.74) 0.67 (0.80) <0.001 <0.001**
Executivne function–Category 0–1 0.98 (0.14) 0.94 (0.25) 0.89 (0.32) 0.002 0.091
Visuospatial–Cube 0–1 0.72 (0.45) 0.48 (0.50) 0.27 (0.45) <0.001 <0.001**
Visuospatial–Grid 0–2 1.98 (0.18) 1.96 (0.25) 1.86 (0.43) 0.002 0.174
Language–Picture naming 0–3 2.89 (0.31) 2.69 (0.54) 2.61 (0.59) <0.001 0.008
Language–Semantic fluency 0–2 1.74 (0.52) 1.34 (0.74) 0.93 (0.86) <0.001 <0.001**
significance ( p < 0.01) after correction for multiple comparisons using the Holm–Bonferroni sequential correction.
** Statistical
Note: The Shapiro–Wilk test of normality demonstrated all VCAT domains and individual test items were not normally distributed ( p < 0.05).
Hence, the Kruskal–Wallis test was performed. For adjusted p-value, multiple linear regression was conducted, controlling for race, age, and
education.

and years of education and adjusting for multiple MCI: 97.4%) and the Grid task (HC: 98.1%, MCI:
comparisons using the Holm–Bonferroni sequential 97.4%), indicative of a ceiling effect (Table 2). This
correction (Table 2). The VCAT items that did not suggests that these two items make smaller
produce significant between-group differences after contributions in terms of distinguishing between
adjustmentsweretheGears,Category,andGriditems. HC and MCI. On the other hand, although floor
effects were not observed, the Gears task did not
Item and domain analysis appear to be useful in separating MCI from mild
The majority of HC and MCI participants achieved AD, as both groups performed equally well on this
perfect scores on the Scene Recall task (HC: 98.6%, task (mean = 1.65).

https://doi.org/10.1017/S1041610219000504 Published online by Cambridge University Press

 Construct validity of VCAT 145

Table 3. VCAT interdomain and item-domain Pearson correlation coefficients

VCAT DOMAINS

EPISODIC ATTENTION/ EXECUTIVE VCAT

MEMORY WM FUNCTION VISUOSPATIAL LANGUAGE TOTAL
...................................................................................................................................................................................................................................................................................................................................

VCAT domains
Episodic memory 1 0.43** 0.48** 0.39** 0.57** 0.88**
Attention/WM 1 0.42** 0.35** 0.38** 0.65**
Executive function 1 0.48** 0.42** 0.73**
Visuospatial 1 0.32** 0.57**
Language 1 0.69**
VCAT items
Shapes 0.65** 0.34** 0.38** 0.31** 0.38** 0.62**
Objects 0.96** 0.42** 0.43** 0.35** 0.55** 0.86**
Scene recall 0.69** 0.29** 0.30** 0.24** 0.31** 0.59**
Shape cancellation 0.45** 1.00** 0.40** 0.28** 0.39** 0.66**
Gears 0.26** 0.22** 0.86** 0.37** 0.21** 0.49**
Patterns 0.55** 0.47** 0.78** 0.46** 0.46** 0.74**
Category 0.21** 0.18** 0.32** 0.27** 0.12** 0.28**
Cube 0.31** 0.28** 0.44** 0.84** 0.26** 0.47**
Grid 0.25** 0.12* 0.28** 0.61** 0.20** 0.33**
Picture naming 0.23** 0.20** 0.31** 0.20** 0.60** 0.37**
Semantic fluency 0.56** 0.37** 0.30** 0.28** 0.89** 0.64**

Note: Shaded figures highlight the corresponding items in each VCAT domain.
* p < 0.05.
** p < 0.01.

Item-domain correlations showed that all VCAT Function and NPA Executive Function, r = 0.47,
items showed a significantly higher correlation with p < 0.001; VCAT Visuospatial and NPA Visuospa-
their respective domains, compared to other do- tial, r = 0.43, p < 0.001; and VCAT Language and
mains (Table 3). All VCAT domains were signifi- NPA Language, r = 0.41, p < 0.001. The VCAT
cantly correlated to each other (p < 0.001), with Episodic Memory was most strongly associated
interdomain Pearson correlation coefficients rang- with Visual Memory (r = 0.67, p < 0.001), followed
ing from 0.32 to 0.57. The VCAT domains of by Visual-Auditory Memory (r = 0.55 p < 0.001)
Episodic Memory and Language produced the high- and Auditory Memory (r = 0.53, p < 0.001).
est correlation (r = 0.57), while the weakest correla- Discriminant validity was also demonstrated for
tion was observed between the Language and the domains of Episodic Memory, Executive Func-
Visuospatial domains (r = 0.32) and between the tion, and Visuospatial, which achieved stronger
Visuospatial and Attention/WM domains (r = 0.35). correlations with their respective NPA domains,
Furthermore, each VCAT domain was more compared to correlations with other NPA domains.
strongly correlated with the total VCAT score than The VCAT domain of Attention/WM was most
with any other domain, which is suggestive of the strongly related to the NPA domain of Attention
discriminative power of the domains. (r = 0.36, p < 0.001) and Executive Functioning
(r = 0.38, p < 0.001), while the VCAT Language
Construct validity domain was most closely related to the NPA domain
In the overall sample, all VCAT domains were of Language (r = 0.41, p < 0.001) and Executive
significantly correlated to the domains created Functioning (r = 0.43, p < 0.001).
from the rest of the NPA battery. Convergent valid- To evaluate the effects of participant character-
ity was demonstrated by the significant correlations istics on VCAT scores in the present cohort, a
between the VCAT domains and their respective multiple regression analysis was conducted, regres-
NPA domains (Table 4). The Pearson correlation sing gender, age, race, education, and language on
coefficients for each pair are as such: VCAT Epi- the total VCAT score in a single regression model to
sodic Memory and NPA Episodic Memory, investigate the effect of each demographic variable
r = 0.61, p < 0.001; VCAT Attention and NPA on VCAT score, controlling for each other. This
Attention, r = 0.36, p < 0.001; VCAT Executive analysis demonstrated that the total VCAT score

https://doi.org/10.1017/S1041610219000504 Published online by Cambridge University Press

 146 A. Low et al.

Table 4. Construct validity—correlational matrix comparing VCAT domains against other neuropsychological
assessment (NPA) domains
VCAT DOMAINS

VCAT EPISODIC ATTENTION/ EXECUTIVE

TOTAL MEMORY WM FUNCTION VISUOSPATIAL LANGUAGE
...................................................................................................................................................................................................................................................................................................................................

NPA domains
Global cognition 0.75** 0.69** 0.42** 0.55** 0.40** 0.51**
MMSE 0.77** 0.72** 0.45** 0.55** 0.45** 0.58**
MoCA 0.81** 0.76** 0.47** 0.58** 0.47** 0.61**
Episodic memory 0.63** 0.61** 0.30** 0.34** 0.26** 0.39**
Attention/WM 0.50** 0.35** 0.36** 0.36** 0.35** 0.35**
Executive function 0.63** 0.54** 0.38** 0.47** 0.35** 0.43**
Visuospatial 0.50** 0.39** 0.30** 0.35** 0.43** 0.21*
Language 0.44** 0.33** 0.20** 0.36** 0.24** 0.41**

Note: Figures represent Pearson’s correlation coefficient.

Shaded figures highlight the correspondent VCAT and NPA cognitive domains.
* p < 0.05.
** p < 0.01.

was not significantly affected by language of admin- Semantic Fluency language measure (r = 0.52).
istration, or participants’ gender, age, race, or edu- Lower inter-item correlations were observed between
cational attainment in HCs and participants with the Shapes and Category items (r = 0.08), the Cube
mild AD (p > 0.05), although the expected age and Grid items (r = 0.08), the Scene and Category
(p < 0.05) and education (p < 0.01) effects persisted item (r = 0.10), and the Scene and Picture Naming
in MCI participants. items (r = 0.08).

Reliability Diagnostic accuracy

The VCAT achieved good internal consistency, ROC analysis (Fig. 1) found that the VCAT was
obtaining a strong Cronbach alpha of 0.74. Psycho- more effective than both the MMSE and MoCA at
metricians recommend a minimum Cronbach’s alpha discriminating between MCI and mild AD partici-
coefficient of 0.65 to 0.8 (Tavakol and Dennick, pants. The VCAT achieved an AUC of 84.2% (95%
2011) and a maximum of 0.9 (Streiner, 2003). Fur- CI = 79.1%–89.3%), while the MMSE and MoCA
ther analysis found that there would be no improve- achieved an AUC of 76.5% (95% CI = 70.4%–
ment in Cronbach’s alpha with the exclusion of any 82.7%) and 81.8% (95% CI = 76.4%–87.2%),
item in the scale, except with regard to the executive respectively.
function item of Category (Cronbach’s alpha if item In terms of detecting cognitive impairment
deleted = 0.74). However, as the importance of the (CI) from HC, the VCAT (AUC = 85.5%, 95%
item outweighs the marginal increase in reliability, we CI = 81.7%–89.3%) performed similarly to the
made the decision to retain it. MoCA (AUC = 85.3%, 95% CI = 81.4%–89.3%)
The correlation of each VCAT item with each and had a higher AUC compared to the MMSE
other was also explored. All 11 items were signifi- (AUC = 79.5%, 95% CI = 75.0%–84.1%).
cantly correlated to each other (p < 0.05), with the
exception of the Shape item and Category item pair Optimal VCAT cutoff scores
(r = 0.08, p = 0.054). The majority of the inter-item The optimal VCAT cutoff to detect CI (MCI+AD)
correlations fell within the ideal range of 0.15 and was ≤ 24. This cutoff provided a specificity of 71.1%
0.50 (Clark and Watson, 1995). The Objects item (95% CI: 70.9%–71.3%) and a sensitivity of 75.4%
had a relatively high inter-item correlation with the (95% CI: 75.3%–75.6%). At this cutoff, the PPV was
other memory items, Shapes (r = 0.44) and Scene 74.3% (95% CI: 74.1%–74.5%), NPV was 72.3%
(r = 0.59). Typically, a coefficient above 0.50 sug- (95% CI: 72.1%–72.5%), PLR was 2.89 (95% CI:
gests item redundancy. However, these higher cor- 2.87–2.92), and NLR was 0.38 (95% CI: 0.38–0.39).
relations were not unexpected, as narrow constructs To differentiate patients with mild AD from those
tend to result in higher inter-item correlations. The with MCI, a cutoff of ≤ 19 was found to be optimal.
Objects item was also highly correlated to the execu- This cutoff provided a sensitivity of 68.3% (95% CI:
tive function measure Pattern (r = 0.50), and the 68.1%–68.6%) and a specificity of 84.8% (95% CI:

https://doi.org/10.1017/S1041610219000504 Published online by Cambridge University Press

 Construct validity of VCAT 147

Figure 1. Receiver Operating Characteristic (ROC) curves: (A) ROC curve separating healthy comparisons (HCs; n = 233) from participants
with cognitive impairment (MCI+AD; n = 238); and (B) ROC curve separating MCI (n = 117) and mild AD (n = 121).

84.6%–85.0%). At this cutoff, the PPV was 82.8% it is not theoretically expected to be related to
(95% CI: 82.6%–83.1%), NPV was 71.4% (95% (discriminant/divergent validity) (Cronbach and
CI: 71.2%–71.7%), PLR was 4.50 (95% CI: Meehl, 1955). The VCAT, and each individual
4.43–4.57), and NLR was 0.37 (95% CI: 0.37–0.38). cognitive subscale, demonstrated good construct
validity in terms of both convergent and divergent
validity.
Discussion Convergent validity was established by the signif-
The findings from this study demonstrated that the icant correlations between the VCAT subdomains
VCAT has good construct validity, by providing and their respective cognitive domains as measured
evidence of convergent and discriminant validity. by well-established cognitive tests from their respec-
The test items in the VCAT also demonstrated good tive domains. Divergent validity was also demon-
internal consistency. ROC analysis suggests that the strated, as each VCAT cognitive domain was more
VCAT is at least equivalent to the MoCA and strongly associated to their respective NPA cognitive
MMSE at distinguishing mild AD from MCI, and domains, compared to others. Although most
between HCs and cognitively impaired individuals. VCAT domains were also significantly associated
Our findings suggest the following cutoffs: Demen- with other NPA domains, this was expected due to
tia 0–19, MCI 20–24, Normal 25–30. the close associations inherent between the various
Internal consistency is a measure of reliability cognitive domains. Notably, although VCAT Atten-
based on the associations between individual test tion was most related to the NPA domains of Atten-
items and is achieved when test items proposing to tion and Executive Functions, this was not
measure the same construct produce similar scores. surprising, given that the two domains have been
The VCAT displayed good internal consistency, commonly found to overlap, with many neuropsy-
achieving a Cronbach’s alpha of 0.74, which is within chological tests used to measure both domains
the recommended range of 0.65 to 0.90 (Streiner, interchangeably. The VCAT Language domain
2003; Tavakol and Dennick, 2011). The majority of was most strongly associated with the cognitive
the inter-item correlations were also within the ideal domains of Language, as expected, but also Execu-
range of 0.15 to 0.50 (Clark and Watson, 1995). tive Function. This might suggest that the VCAT
Construct validity is one of the most important Language domain may also elicit some higher-order
attributes of any test of measure, and is an evaluation executive functions. This may be due to the inclu-
of whether a test and its individual subdomains sion of the Semantic Fluency task, which has been
measure what they were designed to measure. Con- argued to be both a measure of language as well as
struct validity is supported: (1) when the cognitive executive function (Henry and Crawford, 2004;
domains measured in the developed test are strongly Whiteside et al., 2015). Another explanation might
related to the cognitive subdomains hypothesized be the general conflation of executive function
(convergent validity), and (2) when the items are not across all VCAT domains, as observed by the rela-
related, or are less strongly related, to other domains tively high correlation across the NPA executive

https://doi.org/10.1017/S1041610219000504 Published online by Cambridge University Press

 148 A. Low et al.

function domain. This may be due to the nature of to investigate the influence of learning effects on
executive tasks used, which inherently employ other the VCAT.
non-executive cognitive operations.
Given that the VCAT was designed with the
intention of being a language-neutral cognitive Conclusion
assessment tool, we assessed the effects of partici- The VCAT is a reliable and valid, cross-cultural,
pant characteristics on VCAT scores in our present language-neutral cognitive screening tool, equiva-
cohort. In healthy elderly and mild AD participants, lent to the MMSE and MoCA in diagnostic ability.
performance on the VCAT was not affected by The VCAT will allow for meaningful cross-cultural
language of administration, or the participants’ gen- comparisons across countries, without the need for
der, age, race, and educational attainment. How- translations or adaptations. This has important im-
ever, the expected effects of age and education on plications for research, in terms of international
cognition were observed in relation to the VCAT collaborations and clinical trials, as well as clinically,
scores of MCI participants. Similarly, previous in- in populations with diverse multiethnic populations
vestigations have also established that performance such as Southeast Asia.
on the VCAT is not significantly influenced by
gender, age, or educational attainment (Kandiah
et al., 2015) and unaffected by language of adminis- Conflict of interest declaration
tration (Lim et al., 2018).
None.
The VCAT and each of its subdomains were able
to distinguish between HCs, MCI, and mild AD.
Most individual items were also able to distinguish Funding
the three groups, with exception of the Gears, Cate-
gory, and Grid tasks. We observed a ceiling effect for The research was supported by the National Neu-
the Category and Grid tasks, suggesting that the roscience Institute (Singapore) in the design and
tasks are too simple. On the other hand, although conduct of the study. This study was funded by the
HCs performed better on the Gears task than the Singhealth Foundation Grant (Grant Number:
cognitively impaired, MCI and mild AD partici- SHF/FG483P/2012).
pants performed similarly on this task.
ROC analysis was also conducted to compare the
diagnostic capabilities of VCAT against the most Description of authors’ roles
widely used screening tests, the MMSE and MoCA.
In terms of distinguishing between CI and HCs, and Audrey Low contributed to the study design,
betweenMCIand mild AD,the VCAT wason parwith statistical analysis, interpretation of the data, draft-
the MMSEandMoCA. Inbothcases, VCAT achieved ing, and revising the manuscript for intellectual
a good AUC of about 85%. content. Levinia Lim contributed to statistical
ROC analysis also suggests the following cutoffs: analysis, data collection, and the revising of the
Dementia 0–19, MCI 20–24, Normal 25–30. This is manuscript for intellectual content. Linda Lim
slightly different from previously suggested cutoffs contributed to data collection and the revising of
(Dementia 0–17, MCI 18–22, Normal 23–30), per- the manuscript for intellectual content. Benjamin
haps due to differences in education levels in the Wong contributed to data collection and the revising
initial study, whereby the MCI cohort had a lower of the manuscript for intellectual content. Eveline
mean years of education while the AD cohort had a Silva contributed to data collection and the revising
higher mean years of education compared to the of the manuscript for intellectual content. Kok
present cohort. Pin Ng contributed to statistical analysis and the
Limitations of the current study include the lack revising of the manuscript for intellectual content.
of participants with more severe stages of AD and Nagaendran Kandiah contributed to the study
the implied homogeneity of the sample. Future work design, statistical analysis, interpretation of the
should also assess the VCAT’s ability to detect data, drafting, and final approval of the manuscript.
various forms of dementia, such as vascular
dementia, and MCI of multiple etiologies. Further- References
more, as neuropsychological testing (including
VCAT) was conducted in the same session, results Albert, M. S. et al. (2011). The diagnosis of mild cognitive
may be influenced by learning effects. Future work impairment due to Alzheimer’s disease: Recommendations
should evaluate the VCAT on a separate day from from the National Institute on Aging-Alzheimer’s
the MMSE and MoCA to avoid learning effects, and Association workgroups on diagnostic guidelines for

https://doi.org/10.1017/S1041610219000504 Published online by Cambridge University Press

 Construct validity of VCAT 149

Alzheimer’s disease. Alzheimer’s & Dementia, 7, 270–279. Geriatric Medicine, 20, 93–119. doi: 10.1016/j.cger.2003
doi: 10.1016/j.jalz.2011.03.008. .10.004.
Boone, K. B. et al. (2007). The association between McKhann, G. M. et al. (2011). The diagnosis of dementia
neuropsychological scores and ethnicity, language, and due to Alzheimer’s disease: recommendations from the
acculturation variables in a large patient population. National Institute on Aging-Alzheimer’s Association
Archives of Clinical Neuropsychology, 22(3), 355–365. workgroups on diagnostic guidelines for Alzheimer’s
Brickman, A. M., Cabo, R. and Manly, J. J. (2006). disease. Alzheimer’s & Dementia, 7, 263–269. doi: 10.1016/
Ethical issues in cross-cultural neuropsychology. Applied j.jalz.2011.03.005.
Neuropsychology, 13, 91–100. doi: 10.1207/ Morris, J. C. (1993). The Clinical Dementia Rating (CDR):
s15324826an1302_4. current version and scoring rules. Neurology, 43, 2412–2414.
Clark, L. A. and Watson, D. (1995). Constructing doi: 10.1212/wnl.43.11.2412-a.
validity: basic issues in objective scale development. Ní Chaoimh, D. et al. (2015). Importance of different
Psychological Assessment, 7, 309–319. doi: 10.1037// language versions of cognitive screening tests: comparison of
1040-3590.7.3.309. Irish and English versions of the MMSE in bilingual Irish
Cronbach, L. J. and Meehl, P. E. (1955). Construct patients. European Geriatric Medicine, 6, 551–553. doi: 10
validity in psychological tests. Psychological Bulletin, 52, .1016/j.eurger.2015.10.006.
281–302. doi: 10.1037/h0040957. Prince, M. et al. (2013). The global prevalence of dementia:
Franzen, S. et al. (2017). Cross-cultural neuropsychological a systematic review and metaanalysis. Alzheimer’s &
assessment of dementia: directions for test development. Dementia, 9, 63–75. doi: 10.1016/j.jalz.2012.11.007.
Alzheimer’s & Dementia, 13, 1141–1142. doi: 10.1016/j Romero, H. R. et al. (2009). Challenges in the
.jalz.2017.06.1666. neuropsychological assessment of ethnic minorities: summit
Gibbons, L. E. et al. (2002). Cross-cultural comparison of proceedings. The Clinical Neuropsychologist, 23, 761–779.
the mini-mental state examination in United Kingdom and doi: 10.1080/13854040902881958.
United States participants with Alzheimer’s disease. Shah, A., Oommen, G. and Wuntakal, B. (2008). Cross-
International Journal of Geriatric Psychiatry, 17, 723–728. cultural aspects of dementia. Psychiatry, 7, 94–97. doi: 10
doi: 10.1002/gps.683. .1016/j.mppsy.2007.11.014.
Henry, J. D. and Crawford, J. R. (2004). A meta-analytic Sperber, A. D. (2004). Translation and validation of study
review of verbal fluency performance following focal cortical instruments for cross-cultural research. Gastroenterology,
lesions. Neuropsychology, 18, 284–295. doi: 10.1037/0894- 126, S124–S128. doi: 10.1053/j.gastro.2003.10.016.
4105.18.2.284. Stigler, J. W., Lee, S.-Y. and Stevenson, H. W. (1986).
Hoosain, R. (1982). Correlation between pronunciation Digit memory in Chinese and English: evidence for a
speed and digit span size. Perceptual and Motor Skills, 55, temporally limited store. Cognition, 23, 1–20. doi: 10.1016/
1128. doi: 10.2466/pms.1982.55.3f.1128. 0010-0277(86)90051-x.
Kandiah, N. et al. (2015). Early detection of dementia in Streiner, D. L. (2003). Starting at the beginning: an
multilingual populations: Visual Cognitive Assessment Test introduction to coefficient alpha and internal consistency.
(VCAT). Journal of Neurology, Neurosurgery & Psychiatry, Journal of Personality Assessment, 80, 99–103. doi: 10.1207/
87, 156–160. doi: 10.1136/jnnp-2014-309647. s15327752jpa8001_18.
Lim, L. et al. (2018). A novel language-neutral Visual Tavakol, M. and Dennick, R. (2011). Making sense of
Cognitive Assessment Test (VCAT): validation in four Cronbach’s alpha. International Journal of Medical Education,
Southeast Asian countries. Alzheimer’s Research & 2, 53–55. doi: 10.5116/ijme.4dfb.8dfd.
Therapy, 10, 6. doi: 10.1186/s13195-017-0333-z. Teng, E. L. (2002). Cultural and educational factors in the
Loewenstein, D. A. et al. (1994). Potential cultural bias in diagnosis of dementia. Alzheimer Disease & Associated
the neuropsychological assessment of the older adult. Disorders, 16, S77–S79. doi: 10.1097/00002093-
Journal of Clinical and Experimental Neuropsychology, 16, 200200002-00007.
623–629. doi: 10.1080/01688639408402673. Ting, S. K. S. et al. (2013). Digit span: a comparison of
Manly, J. J. (2005). Advantages and disadvantages of Chinese versus alphabetic language speakers in dysexecutive
separate norms for African Americans. The Clinical dementia patients. Journal of Neurology, Neurosurgery &
Neuropsychologist, 19, 270–275. doi: 10.1080/ Psychiatry, 85, 117–118. doi: 10.1136/jnnp-2013-305828.
13854040590945346. Whiteside, D. M. et al. (2015). Verbal fluency: language or
Manly, J. J. and Espino, D. V (2004). Cultural influences executive function measure? Applied Neuropsychology: Adult,
on dementia recognition and management. Clinics in 23, 29–34. doi: 10.1080/23279095.2015.1004574.

https://doi.org/10.1017/S1041610219000504 Published online by Cambridge University Press