Behaviour & Information Technology

ISSN: (Print) (Online) Journal homepage: https://www.tandfonline.com/loi/tbit20

Measuring user competence in using artificial

intelligence: validity and reliability of artificial
intelligence literacy scale

Bingcheng Wang, Pei-Luen Patrick Rau & Tianyi Yuan

To cite this article: Bingcheng Wang, Pei-Luen Patrick Rau & Tianyi Yuan (2022): Measuring
user competence in using artificial intelligence: validity and reliability of artificial intelligence
literacy scale, Behaviour & Information Technology, DOI: 10.1080/0144929X.2022.2072768

To link to this article: https://doi.org/10.1080/0144929X.2022.2072768

Published online: 10 May 2022.

Submit your article to this journal

Article views: 1071

View related articles

View Crossmark data

Full Terms & Conditions of access and use can be found at

https://www.tandfonline.com/action/journalInformation?journalCode=tbit20
BEHAVIOUR & INFORMATION TECHNOLOGY
https://doi.org/10.1080/0144929X.2022.2072768

RESEARCH ARTICLE

Measuring user competence in using artificial intelligence: validity and reliability

of artificial intelligence literacy scale
Bingcheng Wang , Pei-Luen Patrick Rau and Tianyi Yuan
Department of Industrial Engineering, Tsinghua University, Beijing, People’s Republic of China

ABSTRACT ARTICLE HISTORY

As artificial intelligence (AI) became a part of daily life, it has become important to determine user Received 23 June 2021
competence in using AI technology. Here, we propose the concept of AI literacy and develop a Accepted 25 April 2022
quantitative scale for obtaining accurate data regarding the AI literacy of ordinary users. We first
KEYWORDS
identified the primary core constructs of AI literacy, including awareness, use, evaluation, and User competence; model
ethics. Next, we generated 65 items to capture these four constructs; only 31 items were validation; information
retained after a three-step content validation process. Then, we conducted a survey, and literacy; twenty-first century
collected two samples of data. By reducing the number of items using the first sample and abilities; evaluation
performing reliability and validity tests on the second sample, we obtained a 12-item methodologies
instrument for the quantitative measurement of AI literacy. The results confirmed that the
proposed four-construct model is an adequate representation of AI literacy. Further, AI literacy
is significantly related to digital literacy, attitude towards robots, and users’ daily usage of AI.
This study will not only aid researchers in understanding how user competence in using AI
technology affects human–AI interactions but will also help designers develop AI applications
that are aligned with the AI literacy levels of the target users.

1. Introduction
Mutka 2011; Calvani et al. 2008), AI literacy does not
Our lives have changed significantly with the emergence require people to become experts in the underlying the-
of artificial intelligence (AI) technology. With an ory of and developments related to AI. Instead, a person
increasing number of smart devices and AI-embedded would be viewed as being AI literate if they are capable
applications becoming available, we are witnessing an of using AI products proficiently and reasonably.
era where ordinary users are transforming from AI Research on AI literacy is crucial for three main
immigrants into AI natives. The competence required reasons. To begin with, AI literacy will shed light on
to survive and engage in the so-called ‘AI era’ (Daven- the ongoing research in HAII, because one’s literacy
port and Ronanki 2018) is increasingly becoming a of a product can shape one’s mental model of it (Brandt
necessary skill. Several researchers have suggested that 2001; Grassian and Kaplowitz 2001; Rosling and Little-
there is an urgent need to improve people’s ability to more 2011); this is important as mental models play a
use AI in the coming era (Kandlhofer et al. 2016; Su vital role in interaction processes (Norman 2013). It
2018; Tarafdar, Beath, and Ross 2019). At the same may also explain the variations in people’s behaviour
time, others (Jarrahi 2018; Stembert and Harbers when they interact with AI. Second, AI literacy can
2019) have highlighted the positive effects of high AI help quantify user competence in AI usage. Many pre-
competence on human–AI interactions (HAII). vious studies on HAII (Lee and Choi 2017; Luo et al.
Although Long and Magerko (2020) has summarised a 2019; Metelskaia et al. 2018) have examined the partici-
set of core competencies in using AI technology, there pants’ AI competence by measuring their prior experi-
neither exists a mature framework nor a practical tool ence and usage frequency, which cannot be considered
for measuring it. To fill this gap, we propose the concept complete or accurate measurements because they do
of AI literacy to describe the competence of individuals not represent the participants’ full competence
in using AI technology. AI literacy refers to the ability to (Munro et al. 1997) with respect to AI usage. Although
properly identify, use, and evaluate AI-related products prior experience with AI may be closely related to AI lit-
under the premise of ethical standards. Like many other eracy, the development of a scale for quantifying AI lit-
related literacies, such as computer literacy (Hoffman eracy will allow researchers and designers to obtain a
and Blake 2003; Tobin 1983) and digital literacy (Ala- better understanding of user competence in using AI.

CONTACT Pei-Luen Patrick Rau [email protected] Department of Industrial Engineering, Tsinghua University, Beijing, People’s Republic of China
© 2022 Informa UK Limited, trading as Taylor & Francis Group
2 B. WANG ET AL.

Finally, AI literacy will help improve AI education by the conceptual level, AI has a wide integration with
providing a comprehensive framework that can serve the concepts of cognitive neuroscience and other disci-
as an outline for curriculum design (Kandlhofer et al. plines. AI development involves many moral and ethical
2016). issues.
This study makes the following contributions. Firstly, Nevertheless, for a user to be literate in a technology
we propose a theoretical framework of AI literacy con- requires that they have more than a utilitarian perspec-
sisting of four constructs. Secondly, we develop a tive of it (Kandlhofer et al. 2016; Moore 2011). Thus, it is
reliable and valid self-report instrument for measuring essential to understand the entire interaction process
individuals’ AI literacy using two sample sets. Thirdly, and establish the proper attitude and values towards
we explore the relationship between individuals’ AI lit- AI systems. In this section, we discuss the existing
eracy and their digital literacy, attitude towards robots definition and constructs of AI literacy based on existing
and daily usage of AI products and applications. In research on the topic.
the rest of the paper, we first review the relevant litera- AI literacy is not a novel concept, and several
ture on AI literacy and provide a detailed definition and researchers (Druga et al. 2019; Kandlhofer et al. 2016;
model for it based on an analogy with digital literacy. Xiao and Bie 2019) have previously discussed methods
Then, we develop a tool to measure AI literacy, using for integrating AI education into existing educational
the standard process proposed by Hinkin (1998); this frameworks. Kandlhofer et al. (2016) identified seven
process involves the following steps: item generation, topics in AI literacy: automata, intelligent agents, graphs
content validation, questionnaire administration, item and data structures, sorting, problem solving through
reduction, confirmatory factor analysis, and reliability search, classic planning, and machine learning. How-
and validity testing. Next, we explore the relationships ever, these frameworks are only applicable with respect
between users’ AI literacy and users’ attitude and daily to the cultivation of AI literacy in students within the
usage of AI technology. Finally, we evaluate the suit- education system. For the general population, methods
ability and limitations of the proposed instrument and for measuring user knowledge of AI and the ability to
discuss its potential for use in future research on AI. use it remain to be developed. To establish a concrete
and comprehensive framework, we referred to the
underlying concepts of an intimately related literacy,
2. Literature review
namely, digital literacy. The reason for choosing digital
The term literacy originally referred to the basic skills literacy was that there already exists a significant body of
and knowledge associated with books and printings literature on it, including tested theoretical frameworks,
(McMillan 1996). However, with the rapid development which can aid research on AI literacy. In addition, many
of computer and digital technologies, there have been AI applications and products rely on computer technol-
many attempts to extend the conception of literacy ogy and information and communication technology
beyond its original meaning of reading and writing (ICT) (Smith and Eckroth 2017), with a large number
(Buckingham and Burn 2007). Thus, discussions on lit- of the AI technologies available being integrated with
eracy have been extended to television literacy (Buck- digital applications. Being literate in digital products
ingham 1989), information literacy (Eisenberg, Lowe, currently sometimes also implies the ability to use AI
and Spitzer 2004), digital literacy (Gilster and Glister well.
1997), game literacy (Buckingham and Burn 2007), However, we must mention that digital literacy is not
and media literacy (Livingstone and Van der Graaf a substitute for AI literacy in HAII research. Although
2008). AI belongs to the category of digital technology in a
Artificial intelligence (AI) is a new technological sense, AI is regarded as an interdisciplinary field invol-
science for simulating, extending, and expanding ving computer science, information science, mathemat-
human intelligence. Now, the emergence of AI technol- ics, psychology, sociology, linguistics, philosophy, and
ogy has made the concept of AI literacy necessary for many other subjects (Russell and Norvig 2002). The
describing people’s competence in using this technol- intersection of these disciplines naturally indicates the
ogy. Digital technology is a science and technology differences between AI literacy and digital literacy.
associated with electronic computers, which is a general Besides, from users’ perspective, AI usually has more
concept. AI technology is a branch of computer science biological and social attributes than general digital tech-
concentrating on the essence of intelligence. It attempts nology (Minsky 2007; Poria et al. 2017; Tao and Tan
to produce a new intelligent machine that can respond 2005). For example, when users face a robot, they
in a similar way to human intelligence. On a technical often interact with it through social logic instead of
level, AI technology based on digital technology. At the logic to interact with the machine (Bruce,
 BEHAVIOUR & INFORMATION TECHNOLOGY 3

Nourbakhsh, and Simmons 2002; Vossen, Ham, and and skills. Given these definitions, KSAVE can be
Midden 2010). Such differences in interactions reflect viewed as a general framework that can also be applied
the difference in the mental models of users dealing to AI literacy. In fact, considering the close relationship
with AI and digital technology. These differences between digital technology and AI technology, the
would lead users to have different criteria for evaluating above-described models lay the foundation for AI
AI products than for evaluating digital products. AI lit- literacy.
eracy is not the same as digital literacy. It is inappropri- The meaning of digital literacy has been extended
ate to directly describe AI Literacy by using digital significantly in the past decades, as various user devices
literacy related content. For example, a high school stu- have become ubiquitous (Hoffman and Blake 2003).
dent who has not been exposed to the concept of AI but The concepts of literacy must be further integrated
is skilled in using electronic devices has good digital lit- with this new technology and must also include the
eracy but may have poor AI literacy in some aspects. broader use and social importance of computer technol-
Therefore, the instruments for digital literacy are not ogy. Nowadays, AI is also experiencing a similar boom,
appropriate for measuring users’ AI literacy, but the fra- and its connotations and applications are expanding
mework used to build digital literacy can inform the constantly. Therefore, it is difficult to summarise the
establishment of AI literacy. In the following section, specific skills that an AI-literate person should have.
we will review some important concepts in digital lit- In other words, AI literacy ‘should not be thought of
eracy for a clearer insight into AI literacy. as an isolated set of skills but as an essential strategy
There are many overlaps between AI literacy and for most learning objects’ (Moore 2011). In light of
digital literacy. AI literacy and digital literacy share this idea, the technological–cognitive–ethical model
many similarities at the user level. The way people inter- and KSAVE model are more appropriate for AI literacy
act with computers is relatively consistent. Therefore, because they are general frameworks and do not involve
digital literacy can still provide us with many references. specific digital skills. Based on these models, we define
Digital literacy comprises the complex and integrated AI literacy as the ability to be aware of and comprehend
subdisciplines of skill, knowledge, ethics, and creative AI technology in practical applications; to be able to
outputs in the digital environment (Calvani et al. apply and exploit AI technology for accomplishing
2008). Eshet (2004) has suggested that digital literacy tasks proficiently; and to be able to analyze, select, and
refers not only to the ability to use digital devices and critically evaluate the data and information provided
software but also to the cognitive, motor, sociological, by AI, while fostering awareness of one’s own personal
and emotional skills required to work efficiently in a responsibilities and respect for reciprocal rights and
digital environment. Eshet (2004) proposed five dimen- obligations. As per this definition, the following four
sions for digital literacy: photo-visual literacy, reproduc- constructs can be identified for AI literacy: awareness,
tion literacy, branching literacy, informational literacy, usage, evaluation, and ethics. In addition, many litera-
and socioemotional literacy. Gapski (2007) has argued cies have suggested close relationships between users’
that digital literacy includes the following three areas digital literacy and their attitude towards digital tech-
of ability: interpreting information, selecting infor-
nology (Comunello et al. 2015; Eshet 2004; Prior et al.
mation, and articulating information. In turn, these
2016) and their behaviours when using digital technol-
capabilities are suggestive of technological, cognitive,
ogy (Neves, Amaro, and Fonseca 2013; Noh 2017; Park
and ethical proficiencies, which, according to Calvani,
2013). Thus, we can reasonably hypothesise that AI lit-
Fini, and Ranieri (2009), are the three primary aspects
eracy will also be positively related to their attitude
of digital literacy. Ferrari (2012) believes that digital lit-
towards AI and daily usage of AI applications and
eracy lies at the intersection of Internet literacy, infor-
products.
mation communication technology literacy, media
literacy, and information literacy. Wilson, Scalise, and H1a: Users’ AI literacy will be positively correlated with
Gochyyev (2015) proposed an ICT framework called their digital literacy.
KSAVE, which is an acronym for knowledge (K), skills H1b: Users’ AI literacy will be positively correlated with
(S), attitudes (A), values (V), and ethics (E). In KSAVE, their attitude towards AI.
knowledge refers to the specific requirements for under-
standing, such as declarative knowledge and facts; skills H1c: Users’ AI literacy will be positively correlated with
their daily usage of AI applications and products.
are the abilities and processes required to employ and
deploy this knowledge. Further, attitudes, values, and Awareness refers to the ability to identify and compre-
ethics together constitute the behaviours and affective hend AI technology during the use of AI-related appli-
aspects that a person exhibits with respect to knowledge cations. Many researchers (Calvani, Fini, and Ranieri
4 B. WANG ET AL.

2009; Hallaq 2016; Katz 2007; Martin and Grudziecki literacy level from the perspective of use and operation.
2006) have introduced awareness as a cognitive process That is to say, usage in AI literacy will be closely related
that occurs before one uses a particular technology. Hal- to users’ daily usage of AI products and applications.
laq (2016) introduced awareness as one of the five core Also, some researchers (Donat, Brandtweiner, and Ker-
constructs of media literacy. Calvani, Fini, and Ranieri schbaum 2009; Porter and Donthu 2006) have found
(2009) have indicated that the cognitive dimension is that users who use digital products proficiently would
one of the three critical dimensions of digital compe- hold a positive attitude towards digital technology,
tence. In the digital literacy framework called DigEuLit which implies that usage in AI literacy will be positively
(Martin and Grudziecki 2006), awareness includes state- related to users’ attitude.
ment and identification, which are the first two steps of a
H3a: Users’ usage in AI literacy will be positively corre-
13-step process for digital literacy. Another information lated with their attitude towards AI.
literacy framework, called iSkills (Katz 2007), also con-
tains a ‘define’ dimension, which refers to the ability to H3b: Users’ usage in AI literacy will be positively corre-
understand and articulate the scope of an information lated with their daily usage of AI applications and
products.
problem. The above-mentioned concepts attempt to
measure the degree of awareness of the users and under- Evaluation refers to the ability to analyze, select, and
stand their respective literacy levels. Given these facts, critically evaluate AI applications and their outcomes.
we believe that awareness is also an important factor Owing to the complexity and black-box nature of AI
with respect to AI literacy. Some research (Kim 2013; (Mueller et al. 2019), the results produced by it require
Weisberg 2011) indicated strong relationship between careful consideration and evaluation. Thus, evaluation
awareness and attitude. Donat, Brandtweiner, and Ker- is an essential competency with respect to AI literacy.
schbaum (2009) also argued that a positive attitude Evaluation also plays a critical role in other related litera-
would contribute to people’s willingness to use digital cies (Hallaq 2016; Katz 2007; Martin and Grudziecki
products. Thus, we can hypothesise: 2006). For example, DigEuLit (Martin and Grudziecki
2006) includes analysis, evaluation, and interpretation.
H2a: Users’ awareness in AI literacy will be positively
correlated with their attitude towards AI. Further, both iSkills (Katz 2007) and digital online
media literacy (Hallaq 2016) have evaluation in their
H2b: Users’ awareness in AI literacy will be positively models as well. In these models, evaluation often refers
correlated with their daily usage of AI applications to the evaluation of information and results; however,
and products.
in the case of AI literacy, evaluation also requires a user
Usage refers to the ability to apply and exploit AI to form accurate opinions regarding AI applications
technology to accomplish tasks proficiently. This con- and product. In addition, users who are able to evaluate
struct focuses on operational levels, including easy an AI applications or product usually implies rich experi-
access to AI applications and tools, proficiency in the ence of using AI applications or products. Thus, we can
operation of AI applications and tools, and capable inte- hypothesise:
gration of different types of AI applications and tools.
H4a: Users’ evaluation in AI literacy will be positively
Usage belongs to the technological dimension in the correlated with their attitude towards AI.
model of Calvani, Fini, and Ranieri (2009) and is com-
mon to other related literacies as well (Balfe, Sharples, H4b: Users’ evaluation in AI literacy will be positively
and Wilson 2018; Katz 2007; Leahy and Dolan 2010). correlated with their daily usage of AI applications
and products.
For example, the European Computer Driver License
refers to 13 specific skills needed for computer literacy Ethics refers to the ability to be aware of the respon-
(Leahy and Dolan 2010), such as word processing, sibilities and risks associated with the use of AI technol-
spreadsheets, and presentation. iSkills (Katz 2007) also ogy. In the field of AI, ethical issues have always been a
refers to management and information integration for concern of the public. While AI technology brings con-
regular usage. In addition, KSAVE (Wilson, Scalise, venience, it also leads people to think carefully and
and Gochyyev 2015) also contains skills as an essential explicitly about intelligence and ethics within it (Gunkel
aspect of ICT literacy. A few information and media lit- 2012). Therefore, an AI-literate person must be able to
eracy frameworks (Hair et al. 1998; Katz 2007) also correctly understand and judge ethical issues to ensure
include access, which refers to the skills and competen- that AI technology is used correctly and appropriately.
cies needed to locate information and media content In the case of other related literacies as well, many
using the available technologies and associated software. researchers view ethics as one of the critical components
The above-described concepts measure the user’s in their frameworks (Calvani et al. 2008; Hallaq 2016;
 BEHAVIOUR & INFORMATION TECHNOLOGY 5

Wilson and Daugherty 2018). In the digital competence when performing specific AI-involving tasks and
assessment framework (Calvani et al. 2008; Calvani, allowed the respondent to show their degree of agree-
Fini, and Ranieri 2009), the ethical dimension is one ment with them. This format has been shown to be
of the three core dimensions of digital literacy. Hallaq the most effective one because it allows for the possi-
(2016) also introduced ethical awareness in his frame- bility that a respondent may be able to perform a certain
work of media literacy. Wilson, Scalise, and Gochyyev task but chooses not to participate in it (Hallaq 2016).
(2015) regarded attitudes, values, and ethics together To be included, an item had to explicitly capture (at
as the key behaviours and affective aspects related to least) one of the four constructs: awareness, usage,
ICT literacy. Thus, it is reasonable to hypothesise that evaluation, and ethics. The items were also designed
users’ ethics in AI literacy will be positively related to to reflect the literature and to be easy to understand
their attitude towards AI. For daily usage, abuse of the for ordinary people. Besides, to prevent the question-
digital technology usually means low awareness of ethi- naire from becoming obsolete in a short time due to
cal issues, but some research (Bartsch and Dienlin 2016; the rapid development of AI technology, the items
Dinev and Hart 2004) indicates that people who use were narrated from a more general perspective rather
more digital products are more likely to be aware of than asking specific applications. Then the items were
the issues because they are probably the victims of the compared for theoretical deficiency and redundancy
abuse. Therefore, we hypothesise: within each construct. The items that did not meet the
above criteria were eliminated in this step. Finally, we
H5a: Users’ ethics in AI literacy will be positively corre-
lated with their attitude towards AI. obtained a total of 65 items with the number of items
per dimension ranging from 10 to 24.
H5b: Users’ ethics in AI literacy will be positively corre-
lated with their daily usage of AI applications and
products. 3.2. Content validation
After synthesising and analogising several related lit- To measure awareness, usage, evaluation, and ethics, 24,
eracies, we propose a framework for AI literacy with 16, 15, and 10 items were evaluated, respectively. Five
four constructs. These constructs lay the foundation subject matter experts (SMEs) were recruited for this
for the development of the AI Literacy Scale (AILS). part of the study. Three of the SMEs were PhD candi-
In the sections that follow, we will evaluate whether dates in the field of human–computer interactions,
the proposed four-construct framework is the most suit- who had prior experience in scale development and
able one for AI literacy and test the reliability and val- research concerning AI or robotics. The other two
idity of the proposed instrument. We will then test SMEs were PhD candidates in computer science and
our hypotheses to explore the relationships between automation. It should be noted that using a small
user’s AI literacy, attitude, and daily usage. sample of students for content validity assessment was
considered appropriate, since this was a cognitive task
3. Method that did not require an understanding of the phenom-
ena under examination (Anderson and Gerbing 1991;
One of the purposes of this study was to develop a valid
Hinkin 1998; Schriesheim et al. 1993). The definitions
and reliable scale to measure people’s AI literacy for
of the four AI literacy constructs were personally
future HAII research. After receiving approval from
explained to all the SMEs to ensure they understood
the ethics review board of the authors’ university, we
the meaning of each construct. Subsequently, content
developed a standard scale to measure AI literacy.
validation was performed in the following three steps.
AILS was developed following the recommendations
Following the approach proposed by Schriesheim
of Hinkin (1998), and its development included the fol-
et al. (1993), the SMEs were first asked to classify each
lowing steps: item generation, content validation, ques-
item into one of the four constructs. They were also pro-
tionnaire administration, item reduction, confirmatory
vided with an ‘unclassified’ category for items that were
factor analysis, and reliability and validity testing.
deemed to not fit any of the constructs. An item was
considered to clearly address a construct if at least
3.1. Item generation
four of the five SMEs classified it in a similar manner.
To capture the four constructs of AI literacy, we gener- A total of 42 items matched this criterion, with 15
ated self-report items through various methods, such as items being unclassified or classified incorrectly by
heuristic association, brainstorm, and card sorting. The one SME. Further, 23 items were unclassified or
self-report items in AILS were structured as statements classified incorrectly by more than one SME; these
that addressed the behaviour, ability, or level of comfort items were excluded from the rest of the steps.
6 B. WANG ET AL.

Next, we used the approach proposed by Hinkin Internet. However, some researchers (Beach 1989; Pettit
(1985). The SMEs were asked to review the items selected 2002) have pointed out that computer-based surveys are
in the first step and rate them on a three-point Likert scale more likely to result in random responses and nonre-
on the extent to which each item corresponded to the sponses as compared to the paper-and-pencil method.
construct: 1 = no fit, 2 = moderate fit, 3 = good fit. An To counter this disadvantage of electronic surveys, we
item was accepted if at least three SMEs thought it was incorporated an attentiveness question to help the
a ‘good fit’ and no SME thought it was ‘no fit’. Using researchers distinguish those respondents who may
this criterion, 10 items were eliminated, while 31 items have answered the survey questions in a random man-
were selected for the remaining steps. ner. This technique has been employed frequently by
Finally, three of the SMEs were asked to participate in many researchers (Hallaq 2016; Hargittai 2005) to
a focus group to supplement the items (i.e. questions) ensure that they only consider those respondents who
and improve their wording and format. The other two have paid close attention to the survey.
SMEs, who were unable to join the focus group, were We collected two sets of samples, Sample 1, and
interviewed separately by the authors. After the com- Sample 2, through two different methods. Sample 1
pletion of the focus group discussion and interviews, was to be used for item reduction, while Sample 2 was
two items were eliminated, and 14 items were to be used for model validation and confirmation. The
rephrased. We also included an additional item that participants for Sample 1 were recruited through social
had been proposed and approved by all the SMEs. In media and a professional survey company. Of the 684
this manner, a 31-item scale was obtained, wherein participants who completed the online survey, 83 were
nine items are related to AI awareness, nine items to removed for not answering the attentiveness question
AI usage, six items to AI evaluation, and seven items correctly. The final sample size was 601, and the sample
to AI ethics. We ensured that each item was structured was 55.91% female and 44.09% male. The age of the par-
in a manner conducive to obtaining accurate responses. ticipants ranged from 16 to 57, with the mean age being
26.55 (SD = 6.29). Sample 2 was collected by the pro-
fessional survey company. Of the 363 participants who
3.3. Questionnaire administration completed the online survey, 38 were removed for not
AILS is formatted in the form of a seven-point Likert answering the attentiveness question correctly. The
scale. Though both Likert (1932) and Hinkin (1998) final sample size was 325, and the sample was 55.69%
have recommended that new items be scaled using a female and 44.31% male. Further, the age of the partici-
five-point Likert scale, other researchers (Cox 1980; pants ranged from 17 to 65, with the mean age being
Pett, Lackey, and Sullivan 2003) have shown that scales 29.67 (SD = 7.33). As for education, most of the partici-
with five to seven levels are also appropriate. Moreover, pants in Sample 1 (72.21%) and Sample 2 (84.92%) had
a recent study (Finstad 2010) suggested that seven-point a bachelor’s degree. The details of the participants are
scales are more suitable for electronically distributed summarised in Table 1.
usability surveys. Given that our questionnaire is to be In Sample 2, we incorporated three additional
distributed online, we considered a seven-point Likert measures into the questionnaire for examining criterion
scale to be more appropriate. validity and test the hypotheses. The first measure was a
AILS was developed to be presented in an electronic popular digital literacy scale developed by Ng (2012) to
format on computers or smartphones so that it can be measure people’s literacy in information and communi-
transmitted and distributed readily through the cation technology. The scale included six questions for

Table 1. Demographic statistics of the participants in Sample 1 and Sample 2.

Sample 1 Sample 2
Parameter Value/specification Frequency Percentage (%) Frequency Percentage (%)
Gender Female 336 55.91 181 55.69
Male 265 44.09 144 44.31
Age <20 56 9.32 16 4.92
20∼30 372 61.90 148 45.54
30∼40 157 26.12 133 40.92
≥40 16 2.66 28 8.62
Education Middle school 6 1.00 1 .31
High school 54 8.99 10 3.08
Bachelor 434 72.21 276 84.92
Master 97 16.14 36 11.08
Ph. D 10 1.66 2 .62
 BEHAVIOUR & INFORMATION TECHNOLOGY 7

the technical dimension, two for the cognitive dimen- researchers to include, especially when they want to
sion, and two for the social-emotional dimension. For use it for pre-screening or post-task evaluation. For
attitude, we incorporated Negative Attitude towards this purpose, we performed an exploratory factor analy-
Robots Scale (NARS), which measured people’s attitudes sis (EFA). EFA is a technology used to find out the
towards communication robots or agents in daily life essential structure of multivariate observation variables
(Nomura, Kanda, and Suzuki 2006) from the negative and deal with dimensionality reduction. Therefore, EFA
dimension. The reason for choosing NARS was that the can synthesise the variables with complex relationships
scale was one of the most popular instruments in recent into a few core factors. We calculated the item-total cor-
HAII or human robot interaction research field. NARS relations to select the suitable items for the four con-
also consisted of three constructs: attitudes towards situ- structs. All the analyses related to item reduction were
ations of interaction with robots (S1), attitudes towards performed on Sample 1.
social influence of robots (S2), and attitudes towards We first checked the suitability of the data for factor
emotions in interaction with robots (S3). In the third analysis through Kaiser-Meyer-Olkin test, which
measure, we included five questions based on the system confirmed that the sample was suitable for EFA with
usage model of Burton-Jones (2005) to measure the coefficient value of 0.95. The scree plot and parallel analy-
people’s daily usage of AI technology. The five questions sis results as well as the number of eigenvalues greater than
were derived from five aspects: extent of use, variety of 1.00 suggested a four-factor solution, shown in Figure 1.
use, frequency of use, proportion of use, and duration Thus, we ran a four-factor EFA with the diagonal
of use. These three measures would not only serve as cri- rotation and minimum residual estimation. In this
teria for criterion validity analysis but also helped us step, we wanted to ensure that the items matched the
understand the relationship between AI literacy and theoretical structure and to identify any items that
other related factors. were cross loaded in more than one construct. The
results of the EFA are shown in Table 2.
Next, we used confirmation factor analysis (CFA).
4. Results and analysis Different from EFA, CFA is a process of using sample
data to verify the factor structure hypothesis made
4.1. Item reduction
according to some theories and a priori knowledge. In
The item reduction process aimed to ensure that a fair confirmatory factor analysis, the factor structure estab-
number of items representing the four AILS constructs lished according to the existing theory can form an esti-
was retained, thus providing evidence of construct val- mated covariance matrix. We performed CFA for each
idity (Hinkin 1998). Item reduction can also reduce par- subscale to check the factor loadings for their respective
ticipants’ cognitive load and thus increase the usability constructs. We also determined the item-total corre-
of the scale. We limited the number of items to around lations to check if any items on the scale were inconsist-
10 in our final scale, making it palatable for AI ent with the average behaviour of the others. The item

Figure 1. Scree plot.

8 B. WANG ET AL.

Table 2. Results of exploratory factor analysis.

Item Awa. Usa. Eva. Eth. Item Awa. Usa. Eva. Eth.
AW_1 .65 .24 .17 .27 US_8 .36 .40 .16 .19
AW_2 .54 .11 .15 .18 US_9 .26 .65 .13 .16
AW_3 .45 .20 .23 .15 EV_1 .21 .13 .61 .11
AW_4 .57 .27 .18 .17 EV_2 .07 .14 .73 .21
AW_5 .17 -.06 .16 .17 EV_3 .21 .11 .64 .21
AW_6 .45 .19 .20 .34 EV_4 .15 .16 .61 .21
AW_7 .44 .26 .23 .29 EV_5 .21 .14 .29 .28
AW_8 .58 .23 .15 .23 EV_6 .23 .15 .72 .14
AW_9 .65 .20 .19 .24 ET_1 .27 .27 .15 .72
US_1 .14 .74 .10 .19 ET_2 .30 .24 .25 .64
US_2 .12 .62 .19 .29 ET_3 .29 .20 .19 .46
US_3 .21 .71 .09 .20 ET_4 .03 .06 .06 .19
US_4 .38 .38 .02 .00 ET_5 .30 .28 .17 .62
US_5 .13 .70 .08 .22 ET_6 .08 .13 .12 .44
US_6 .15 .68 .23 .22 ET_7 .30 .16 .19 .48
US_7 .31 0.48 .12 .08
Note. Items in boldface are included in the final scale. Factor loadings less than .30 are suppressed. Awa. = Awareness, Usa. = Usage, Eva. = Evaluation, Eth. =
Ethics. χ2(601) = 483.36, p < .0001, Tucker-Lewis index of factoring reliability is .98, root mean square error of approximation (RMSEA) is .03, root mean square
of the residuals (RSMR) is .02.

loadings and item-total correlation results for each of Table 4. We found that the theoretical model used for
the four constructs are listed in Table 3. We selected AI literacy was an acceptable one (CFI = 0.99, TLI =
the following items: (a) those with factor loadings 0.99, GFI = 0.98, RMSEA = 0.01, SRMR = 0.03). In
higher than 0.30 (Costello and Osborne 2005); (b) addition, we also performed CFAs to test four compet-
those with item-total correlation coefficients higher ing, theoretically meaningful models: a single-factor
than 0.60 (Hair et al. 1998); (c) those with the highest model, where all 12 items are loaded onto one factor;
three item-total correlation coefficients; and (d) those a two-factor model, where the items measuring aware-
that did not exhibit redundancy with respect to the ness and ethics (cognitive level) are loaded onto one fac-
other high-loading and high-item-total-correlation tor while those for usage and evaluation (technological
items. Eventually, twelve items emerged from the 31 level) are loaded onto the other factor; a three-factor
items with three items for each construct. model, where the items measuring awareness (cognitive
level) are loaded onto one factor, those measuring usage
and evaluation (technological level) are loaded onto
4.2. Model validation and comparison another factor, and those measuring ethics (ethical
After finalising the items in the scale, we confirmed level) are loaded onto the third factor; and a four-factor
whether the theoretical model used was the best one model, where the awareness, usage, evaluation, and
for measuring AI literacy. For this, we conducted a ethics items are loaded onto four individual factors.
series of CFAs on Sample 2. The results are shown in The fit statistics for the four models are listed in
Table 4. The results suggest that the four-factor model
exhibited better fit statistics than the other models,
Table 3. Item pool factor loadings and corrected item-total
which indicates that the theoretical model based on
correlations.
Item-total Item-total
four distinct constructs was the most appropriate for
Item Loading correlation Item Loading correlation conceptualizing AI literacy. The descriptions and load-
AW_1 0.77 0.77 US_8 0.54 0.62 ings of the twelve items for the finalised scale are
AW_2 0.58 0.66 US_9 0.72 0.75 shown in Table 5.
AW_3 0.55 0.65 EV_1 0.66 0.73
AW_4 0.66 0.69 EV_2 0.76 0.79
AW_5 0.22 0.37 EV_3 0.71 0.77
AW_6 0.61 0.66 EV_4 0.67 0.73 Table 4. Model fit statistics.
AW_7 0.62 0.68 EV_5 0.41 0.56
AW_8 0.7 0.72 EV_6 0.79 0.81 Factor c2 df p CFI TLI GFI RMSEA SRMR
AW_9 0.76 0.76 ET_1 0.82 0.8 1 factor 397.09 54 < .01 .69 .62 .81 .14 .10
US_1 0.77 0.78 ET_2 0.79 0.79 2 factors 344.76 53 < .01 .74 .68 .82 .13 .09
US_2 0.71 0.72 ET_3 0.59 0.66 3 factors 218.11 51 < .01 .85 .81 .87 .10 .07
US_3 0.77 0.79 ET_4 0.21 0.45 4 factors 49.62 48 .41 .99 .99 .98 .01 .03
US_4 0.46 0.59 ET_5 0.77 0.78 Note. The four-factor model is the theoretical model. CFI = comparative fit
US_5 0.75 0.76 ET_6 0.47 0.56 index; TLI = Tucker-Lewis index; GFI = goodness of fit index; RMSEA =
US_6 0.75 0.76 ET_7 0.53 0.6 root mean square error of approximation; SRMR = standardised root
US_7 0.56 0.65 mean square residual. TLI, CFI and GFI values greater than .900 and
Note. Items in boldface are included in the final scale. RMSEA values less than .050 suggest adequate model fit.
 BEHAVIOUR & INFORMATION TECHNOLOGY 9

Table 5. Descriptions and loadings on a standardised scale. 0.70 and higher are considered acceptable (Hair et al.
Item Description Construct Loading 1998). AVE measures the level of variance captured by
AW_1 I can distinguish between smart devices Awareness .72 a construct versus that attributable to measurement
and non-smart devices.
AW_8 I do not know how AI technology can help Awareness .64 error. Values greater than 0.5 indicate adequate conver-
me. R gence (Hair et al. 1998); thus, the average factor loadings
AW_9 I can identify the AI technology employed Awareness .70
in the applications and products I use.
should be larger than 0.72 to ensure AVE values greater
US_1 I can skilfully use AI applications or Usage .72 than 0.5. However, in some cases, and especially for a
products to help me with my daily work. new instrument, factor loadings higher than 0.5 (i.e.
US_3 It is usually hard for me to learn to use a Usage .66
new AI application or product. R AVE > 0.25) could be considered acceptable (e.g. Hsu
US_5 I can use AI applications or products to Usage .72 and Wu 2013; Hu 2013; Mayerl 2016). In our scale,
improve my work efficiency.
EV_2 I can evaluate the capabilities and Evaluation .71 CR values were higher than 0.7, and AVE values were
limitations of an AI application or close to 0.5, which indicated acceptable convergence.
product after using it for a while.
EV_3 I can choose a proper solution from Evaluation .72 Discriminant validity was measured based on the
various solutions provided by a smart HTMT approach proposed by Henseler, Ringle, and
agent.
EV_6 I can choose the most appropriate AI Evaluation .78
Sarstedt (2015). It suggested that the HTMT should
application or product from a variety for not be higher than 0.85 for high discriminant validity
a particular task. (Clark and Watson 1995; Kline 2015). In the case of
ET_1 I always comply with ethical principles Ethics .76
when using AI applications or products. the proposed scale, the results showed that the HTMT
ET_2 I am never alert to privacy and information Ethics .60 values of all four constructs were lower than the
security issues when using AI
applications or products. R threshold of 0.85, indicating acceptable discriminant
ET_5 I am always alert to the abuse of AI Ethics .73 validity for the instrument. In addition, almost all corre-
technology.
R
lations among the four constructs are between .15 and
indicates that item is in reverse form.
.50, which were also considered ideal according to
Clark and Watson (2016).
4.3. Reliability and construct validity
The reliability and validity of AILS were determined
4.4. Hypothesis test results
using Cronbach’s alpha as well as the composite
reliability (CR), average variance extracted (AVE), and The correlations for hypothesis testing are presented in
heterotrait-monotrait ratio of the correlations Table 7. The result partly proved the Hypothesis H1a.
(HTMT). The results are presented in Table 6. Cron- As we mentioned, AI literacy and digital literacy
bach’s alpha is an index used to measure reliability, might share some overlapping parts. Hypothesis H1b
and a value greater than 0.7 is considered acceptable suggested that the participants’ AI literacy would be
(DeVellis 2016). The value of Cronbach’s alpha for positively correlated with their attitude towards AI.
AILS was 0.83, whereas the alpha values for the four Results suggested that AI literacy was significantly and
constructs were 0.73, 0.75, 0.78, and 0.73, respectively. negatively correlated with NARS, S1, S2, and S3, thus
Though all four constructs exhibited reliabilities of supporting H1b. Hypothesis H1c was partly supported
more than 0.70, the instrument itself scored higher as AI literacy was only significantly and positively corre-
than 0.80, which indicated that the instrument in its lated with the extent, variety, and frequency of use of AI
entirety is more reliable than the separate constructs. technology in daily life. Results supported Hypothesis
According to the criterion established by Fornell and H2a, finding that the awareness was significantly and
Larcker (1981), CR and AVE were used to assess the negatively correlated with NARS and NARS S1.
scale’s convergent validity. CR is a less biased estimate Hypothesis H2b was also partly supported, as the aware-
of reliability than Cronbach’s alpha, and CR values of ness in AILS was significantly and positively correlated

Table 6. Results of reliability and validity analyses of AILS.

Correlation & HTMT
Construct Mean SD PV α CR AVE Awareness Usage Evaluation Ethics
Awareness 5.93 .81 0.08 .73 .73 .48 1 .66 .53 .49
Usage 5.76 .77 0.11 .75 .75 .49 .49 1 .44 .41
Evaluation 5.62 .85 0.10 .78 .78 .55 .40 .34 1 .41
Ethics 5.76 .80 0.09 .73 .73 .49 .36 .30 .31 1
AILS 5.77 .58 .83 .88 .50 .78 .72 .72 .68
Note. PV = proportion variance, α = Cronbach’s alpha, CR = composite reliability, AVE = average variance extracted, HTMT = heterotrait-monotrait ratio of the
correlations. The HTMT results are displayed above the diagonal, and the correlation results are below the diagonal.
10 B. WANG ET AL.

Table 7. Correlation between AILS and criterion factors. To demonstrate the predictive power of the four con-
AILS Awareness Usage Evaluation Ethics structs of AILS, we conducted a multiple regression of
Digital Literacy 0.76* 0.55* 0.59* 0.57* 0.49* participants’ AI literacy on their digital literacy, negative
NARS −0.35* −0.29* −0.24* −0.29* −0.19*
NARS S1 −0.40* −0.36* −0.29* −0.27* −0.23* attitude towards robots, and daily usage of AI technol-
NARS S2 −0.22* −0.17 −0.17 −0.22* −0.09 ogy. The results of the regression are presented in
NARS S3 −0.23* −0.16 −0.11 −0.21* −0.17
Extent of Use 0.35* 0.33* 0.38* 0.23* 0.07
Table 8. The Cronbach’s alpha of digital literacy scale
Variety of Use 0.41* 0.32* 0.44* 0.30* 0.15 was 0.83, but the alpha of the two subscales (cognitive
Frequency of Use 0.31* 0.24* 0.38* 0.24* 0.05 dimension and social-emotional dimension) were far
Proportion of Use 0.18 0.11 0.28* 0.17 −0.03
Duration of Use 0.15 0.08 0.23* 0.13 −0.01 below 0.70 (0.39 and 0.39, respectively). Thus, we only
Note. NARS = Negative Attitude towards Robots Scale. S1 = attitude towards used the digital literacy scale in its entirety for predictive
situations of interaction with robots. S2 = attitude towards social influence criterion. Results suggested that the four constructs of
of robots. S3 = attitude towards emotions in interaction with robots. * rep-
resents p-value < .05. AILS combined explained 58% of the variance in
reported digital literacy. All four constructs turned out
to be significant predictors of digital literacy. For
with the extent, variety, and frequency of use of AI tech- NARS and its subscales, the values of Cronbach’s
nology in daily life. Hypotheses H3a and H3b suggested alpha were 0.88, 0.80, 0.80, and 0.80. The results
that the usage in AI literacy would be positively corre- suggested that the four constructs of AILS combined
lated with attitude and daily use of AI technology, explained 13%, 17%, 6%, and 6%, respectively, of the
respectively, and the results supported both hypotheses. variance in NARS and its subscales. Awareness and
The results indicated that the usage was significantly evaluation were significant negative predictors for
and negatively correlated with NARS and NARS S1. both NARS and NARS S1 (attitudes towards situations
Results also indicated that the usage in AI literacy was of interaction with robots). But for NARS S2 (attitudes
significantly and positively with the extent, variety, fre- towards social influence of robots) and S3 (attitudes
quency, proportion, and duration of use of AI technol- towards emotions in interaction with robots), evalu-
ogy in daily life. Hypothesis H4a was fully supported as ation was the only factor predicting the results.
the evaluation was significantly and negatively corre- In terms of users’ daily usage of AI technology, we
lated with NARS, S1, S2, and S3. Results partly sup- explored its relationship with AILS from five aspects.
ported Hypothesis H4b, finding that the evaluation The four constructs of AILS combined explained 19%
was significant and positively correlated with the extent, of the variance in the extent of use, while awareness,
variety, and frequency of use of AI technology in daily usage, and ethics were significant predictors. The four
life. For hypotheses H5a and H5b, the results only sup- constructs of AILS combined explained 23% of the var-
ported H5a, finding that the ethics was significantly and iance for the variety of use, while usage and evaluation
negatively correlated with NARS and NARS S1. The emerged as significant predictors. Both frequency of
ethics was found to have nothing to do with daily use and proportion of use were determined to have sig-
usage. In addition, the results indicated a close relation- nificant relationship with usage, evaluation, and ethics
ship between AI literacy and digital literacy as digital lit- with explained variances of 17% and 10%. For the dur-
eracy was significantly and positively with AI literacy, ation of usage, usage was the only significant predictor,
awareness, usage, evaluation, and ethics. and the four constructs of AILS explained 7% of its

Table 8. Summary of multiple regression results.

Digital literacy NARS NARS S1 NARS S2 NARS S3
Scale β R2 p β R2 p β R2 p β R2 p β R2 p
Awareness .16 < .001* -.17 .02* -.27 < .001* -.09 .35 -.09 .34
Usage .30 < .001* -.11 .15 -.13 .07+ -.13 .19 .00 .99
Evaluation .27 < .001* -.19 .01* -.13 .04* -.24 < .001* -.22 .01*
Ethics .21 < .001* -.06 .32 -.09 .16 .02 .83 -.14 .10
AILS .58 < .001* .13 < .001* .17 < .001* .06 < .001* .06 < .001*
Scale Extent of Use Variety of Use Frequency of Use Proportion of Use Duration of Use
β R2 p β R2 p β R2 p β R2 p β R2 p
Awareness .35 < .001* .18 .10 .12 .31 -.06 .68 .10 .52
Usage .57 < .001* .68 < .001* .68 < .001* .65 < .001* .62 < .001*
Evaluation .16 .12 .28 .01* .24 .02* .25 .04* .21 .12
Ethics -.21 .05* -.08 .45 -.23 .03* -.32 .01* -.23 .10
AILS .19 < .001* .23 < .001* .17 < .001* .10 < .001* .07 < .001*
Note. NARS = Negative Attitudes towards Robots Scale. S1 = attitudes towards situations of interaction with robots. S2 = attitudes towards social influence of
robots. S3 = attitudes towards emotions in interaction with robots. * represents p-value < 0.05, and + represents p-value < 0.10.
 BEHAVIOUR & INFORMATION TECHNOLOGY 11

variance. Across the five dimensions of daily use, the Several points need to be noted when applying AILS
construct usage emerged as the most significant predic- to use in practice. The first one is that the constructs
tor while other constructs, which share enough alone are less reliable than the instrument as a compo-
explained variance, were below significance as unique site. Though all four constructs exhibited reliabilities
predictors. of more than 0.70, the instrument itself scored higher
than 0.80. The recommendation then is to use the
instrument in its entirety rather than the individual con-
5. Discussion and conclusions
structs. It should also be noted the correlation between
This study aimed to improve our understanding of AI AILS and digital literacy was high, which does not mean
literacy by developing and validating a novel measure that AI literacy and digital literacy can be used inter-
of AI literacy. The proposed scale, namely, AILS, is changeably. As discussed in the literature review, the
based on the conceptually similar idea of digital literacy definitions of AI and ICT are different, and users have
suggested by Balfe, Sharples, and Wilson (2018) and different perceptions of the two technologies, which
Calvani et al. (2008). The theoretical basis for their would lead to different attitudes and use of the two
approach to digital literacy led us to believe that a simi- kinds of products. Thus, it is recommended for
lar framework may be applicable in the case of AI lit- researchers and practitioners to use the AILS for user
eracy as well. Using a six-step approach, we were able research involving AI technology. It is worth noting
to demonstrate the reliability and validity of AILS. that users are likely to mistake AI literacy for compe-
The results of factor analyses indicated that the theoreti- tence of using specific applications, given that AI is
cal model based on four distinct constructs is the most usually embedded in the applications. As user compe-
appropriate conceptualization model for AI literacy. tence regarding different applications varies signifi-
Although the AVE values of some constructs were cantly, it is likely to cause an inconsistency in the
slightly lower than the threshold value of 0.50, the results. Thus, we recommend using AILS to measure
whole scale proved to have sufficient convergence val- users’ general AI competence rather than the ability to
idity. The other indicators, such as CR and HTMT, use specific AI applications. Finally, AI literacy might
also suggested sufficient construct validity. relate to the digital literacy, we recommend using the
In the hypotheses testing, most of the hypotheses outcome to explore the association between AI literacy
have been verified. AI literacy and the four constructs and digital literacy.
were found to be negatively correlated with negative This study has certain limitations. First, this study
attitude towards robots (NARS) and negative attitude does not address all concerns regarding AI literacy,
towards situations of interaction with robots (NARS and much work remains to be done on this topic. For
S1). The extent, variety, and frequency of use of AI tech- instance, future studies should examine other theoreti-
nology were found to be correlated with AI literacy and cally meaningful structures. It is also worth noting
the four constructs except ethics. The results of the cor- that both samples scored highly on AILS, meaning
relation and the regression showed that digital literacy that either the respondents possessed high levels of lit-
and AILS were closely related, which could be intuitively eracy or that the attitude-based nature of the scale,
explained: digital literacy is a prerequisite for AI literacy wherein the respondents report their levels of agree-
since most of today’s AI technology is embedded in ment with the statements regarding AI, fails to accu-
digital products. It might come from the section based rately assess their understanding of AI. It is also
on the digital literacy. The regression results also indi- possible that the respondents who completed the survey
cated that awareness and evaluation in AILS played an online have more experience in using AI applications.
important role in predicting users’ negative attitudes Hence, the relationship between digital literacy and AI
towards robots. This could be attributed to the fact literacy is worth investigating in the future. In addition,
that awareness and evaluation represented users’ cogni- we must recognise the limitations of the SMEs.
tion and judgement of AI, which would affect their atti- Although they had training in the relevant analysis
tudes towards robots in the end. In terms of users’ daily methods and work in related fields, they are PhD stu-
AI usage, the results that the construct of usage in AILS dents. Future studies should examine whether AI
showed a significant relationship with user’s daily researchers have different conceptualizations of the con-
behaviour in using AI technology were not surprising. tent and items included in the measure.
The findings indicated that AILS is an important predic- That said, this study lays the foundation for future
tor of user’s attitude and behaviour towards AI technol- research on the effects of AI literacy on various out-
ogy usage. In general, these results indicated that AILS is comes. We have provided a comprehensive definition
suitable to measure AI literacy. of AI literacy and created a well-defined four-factor
12 B. WANG ET AL.

model. Further, we have developed a short 12-item Bruce, A., I. Nourbakhsh, and R. Simmons. 2002. “The Role of
measure to provide researchers and practitioners with Expressiveness and Attention in Human-Robot
a tool for assessing the self-report competence of users Interaction.” Paper Presented at the Proceedings 2002
IEEE International Conference on Robotics and
in using AI. By defining the AI literacy domain and pro- Automation (Cat. No. 02CH37292).
viding effective measurement tools, we hope to improve Buckingham, D. 1989. “Television Literacy: A Critique.”
and encourage future research in this area. For example, Radical Philosophy 51: 12–25.
one domain that may benefit significantly from this Buckingham, D., and A. Burn. 2007. “Game Literacy in
work is explainable AI. The explanations provided by Theory and Practice.” Journal of Educational Multimedia
and Hypermedia 16 (3): 323–349.
intelligent systems or agents also improve users’ literacy,
Burton-Jones, A. 2005. New Perspectives on the System Usage
leading to a better collaboration with the systems or the Construct.
agents (Zhou, Itoh, and Kitazaki 2021). Moreover, the Calvani, A., A. Cartelli, A. Fini, and M. Ranieri. 2008. “Models
conceptualization of AI literacy and the development and Instruments for Assessing Digital Competence at
of suitable tools for measuring it will help designers to School.” Journal of E-Learning and Knowledge Society 4
portray accurate user model and subsequently design (3): 183–193.
Calvani, A., A. Fini, and M. Ranieri. 2009. “Assessing Digital
appropriate explainable AI systems based on these Competence in Secondary Education. Issues, Models and
models. Finally, because the proposed model and the Instruments.” Issues in Information and Media Literacy:
scale based on it define and capture the constructs of Education, Practice and Pedagogy, 153–172.
AI literacy, researchers will be able to discern more Clark, L. A., and D. Watson. 1995. “Constructing Validity:
nuanced relationships between AI literacy and user Basic Issues in Objective Scale Development.”
Psychological Assessment 7 (3): 309–319.
experience in HAII research.
Clark, L. A., and D. Watson. 2016. Constructing Validity:
Basic Issues in Objective Scale Development.
Disclosure statement Comunello, F., S. Mulargia, F. Belotti, and M. Fernández-
Ardèvol. 2015. “Older People’s Attitude Towards Mobile
No potential conflict of interest was reported by the author(s). Communication in Everyday Life: Digital Literacy and
Domestication Processes.” Paper presented at the
International Conference on Human Aspects of it for the
Funding Aged Population.
Costello, A., and J. Osborne. 2005. “Best Practices in Exploratory
This work was supported by National Key R&D Program of
Factor Analysis: Four Recommendations for Getting the
China (2018AAA0101702).
Most from Your Analysis.” Pract Assess, Res Eval.
Cox III, E. P. 1980. “The Optimal Number of Response
Alternatives for a Scale: A Review.” Journal of Marketing
ORCID Research 17 (4): 407–422.
Bingcheng Wang http://orcid.org/0000-0003-0996-6824 Davenport, T. H., and R. Ronanki. 2018. “Artificial
Pei-Luen Patrick http://orcid.org/0000-0002-5713-8612 Intelligence for the Real World.” Harvard Business Review
Tianyi Yuan http://orcid.org/0000-0002-7134-130X 96 (1): 108–116.
DeVellis, R. F. 2016. Scale Development: Theory and
Applications. Vol. 21. Sage publications.
Dinev, T., and P. Hart. 2004. “Internet Privacy, Social
References
Awareness, and Internet Technical Literacy. An
Ala-Mutka, K. 2011. “Mapping Digital Competence: Towards Exploratory Investigation.” BLED 2004 Proceedings, 24.
a Conceptual Understanding.” Sevilla: Institute for Donat, E., R. Brandtweiner, and J. Kerschbaum. 2009.
Prospective Technological Studies. “Attitudes and the Digital Divide: Attitude Measurement
Anderson, J. C., and D. W. Gerbing. 1991. “Predicting the as Instrument to Predict Internet Usage.” Informing
Performance of Measures in a Confirmatory Factor Science: The International Journal of an Emerging
Analysis with a Pretest Assessment of Their Substantive Transdiscipline 12: 37–56.
Validities.” Journal of Applied Psychology 76 (5): 732. Druga, S., S. T. Vu, E. Likhith, and T. Qiu. 2019. “Inclusive AI
Balfe, N., S. Sharples, and J. R. Wilson. 2018. “Understanding Literacy for Kids Around the World.” Paper Presented at
Is Key: An Analysis of Factors Pertaining to Trust in a Real- the Proceedings of FabLearn 2019.
World Automation System.” Human Factors 60 (4): 477– Eisenberg, M. B., C. A. Lowe, and K. L. Spitzer. 2004.
495. doi:10.1177/0018720818761256. Information Literacy: Essential Skills for the Information
Bartsch, M., and T. Dienlin. 2016. “Control Your Facebook: Age. ERIC.
An Analysis of Online Privacy Literacy.” Computers in Eshet, Y. 2004. “Digital Literacy: A Conceptual Framework for
Human Behavior 56: 147–154. Survival Skills in the Digital era.” Journal of Educational
Beach, D. A. 1989. “Identifying the Random Responder.” The Multimedia and Hypermedia 13 (1): 93–106.
Journal of Psychology 123 (1): 101–103. Ferrari, A. 2012. “Digital Competence in Practice: An Analysis
Brandt, D. S. 2001. Information Technology Literacy: Task of Frameworks.” In: Luxembourg: Publication office of the
Knowledge and Mental Models. EU. Research Report by the Joint … .
 BEHAVIOUR & INFORMATION TECHNOLOGY 13

Finstad, K. 2010. “Response Interpolation and Scale Kline, R. B. 2015. Principles and Practice of Structural
Sensitivity: Evidence Against 5-Point Scales.” Journal of Equation Modeling. Guilford publications.
Usability Studies 5 (3): 104–110. Leahy, D., and D. Dolan. 2010. “Digital Literacy: A Vital
Fornell, C., and D. F. Larcker. 1981. “Evaluating Structural Competence for 2010?” Paper Presented at the IFIP
Equation Models with Unobservable Variables and International Conference on Key Competencies in the
Measurement Error.” Journal of Marketing Research 18 Knowledge Society.
(1): 39–50. Lee, S., and J. Choi. 2017. “Enhancing User Experience with
Gapski, H. 2007. “Some Reflections on Digital Literacy.” Paper Conversational Agent for Movie Recommendation:
Presented at the Proceedings of the 3rd International Effects of Self-Disclosure and Reciprocity.” International
Workshop on Digital Literacy. Journal of Human-Computer Studies 103: 95–105.
Gilster, P., and P. Glister. 1997. Digital Literacy. New York: Likert, R. 1932. “A Technique for the Measurement of
Wiley Computer Pub. Attitudes.” Archives of Psychology.
Grassian, E. S., and J. R. Kaplowitz. 2001. “Information Livingstone, S., and S. Van der Graaf. 2008. “Media Literacy.”
Literacy Instruction.” Neal-Schuman, New York, NY. The International Encyclopedia of Communication.
Gunkel, D. J. 2012. The Machine Question: Critical Long, D., and B. Magerko. 2020. “What is AI Literacy?
Perspectives on AI, Robots, and Ethics. MIT Press. Competencies and Design Considerations.” Paper
Hair, J. F., W. C. Black, B. J. Babin, R. E. Anderson, and R. L. Presented at the CHI ‘20: CHI Conference on Human
Tatham. 1998. Multivariate Data Analysis. 5 vols. Upper Factors in Computing Systems, April 21.
Saddle River, NJ: Prentice hall. Luo, X., S. Tong, Z. Fang, and Z. Qu. 2019. “Machines
Hallaq, T. 2016. “Evaluating Online Media Literacy in Higher Versus Humans: The Impact of AI Chatbot Disclosure on
Education: Validity and Reliability of the Digital Online Customer Purchases.” Luo, X, Tong S, Fang Z, Qu (2019).
Media Literacy Assessment (DOMLA).” Journal of Media Martin, A., and J. Grudziecki. 2006. “DigEuLit: Concepts and
Literacy Education 8 (1): 62–84. Tools for Digital Literacy Development.” Innovation in
Hargittai, E. 2005. “Survey Measures of Web-Oriented Digital Teaching and Learning in Information and Computer
Literacy.” Social Science Computer Review 23 (3): 371–379. Sciences 5 (4): 249–267.
Henseler, J., C. M. Ringle, and M. Sarstedt. 2015. “A New Mayerl, J. 2016. “Environmental Concern in Cross-National
Criterion for Assessing Discriminant Validity in Comparison: Methodological Threats and Measurement
Variance-Based Structural Equation Modeling.” Journal of Equivalence.” In Green European, 210–232. Routledge.
the Academy of Marketing Science 43 (1): 115–135. McMillan, S. 1996. “Literacy and Computer Literacy:
Hinkin, T. R. 1985. “Development and Application of New Definitions and Comparisons.” Computers & Education
Social Power Measures in Superior-Subordinate 27 (3-4): 161–170.
Relationships.” University of Florida. Metelskaia, I., O. Ignatyeva, S. Denef, and T. Samsonowa.
Hinkin, T. R. 1998. “A Brief Tutorial on the Development of 2018. “A Business Model Template for AI Solutions.”
Measures for Use in Survey Questionnaires.” Paper Presented at the Proceedings of the International
Organizational Research Methods 1 (1): 104–121. Conference on Intelligent Science and Technology.
Hoffman, M., and J. Blake. 2003. “Computer Literacy: Today Minsky, M. 2007. The Emotion Machine: Commonsense
and Tomorrow.” Journal of Computing Sciences in Colleges Thinking, Artificial Intelligence, and the Future of the
18 (5): 221–233. Human Mind: Simon and Schuster.
Hsu, L., and P. Wu. 2013. “Electronic-Tablet-Based Menu in a Moore, D. R. 2011. “Technology Literacy: The Extension of
Full Service Restaurant and Customer Satisfaction-a Cognition.” International Journal of Technology and
Structural Equation Model.” International Journal of Design Education 21 (2): 185–193.
Business, Humanities and Technology 3 (2): 61–71. Mueller, S. T., R. R. Hoffman, W. Clancey, A. Emrey, and G.
Hu, C. 2013. “A New Measure for Health Consciousness: Klein. 2019. “Explanation in Human-AI Systems: A
Development of A Health Consciousness Conceptual Literature Meta-Review, Synopsis of Key Ideas and
Model.” Paper presented at the Unpublished paper pre- Publications, and Bibliography for Explainable AI.” arXiv
sented at National Communication Association 99th preprint arXiv:1902.01876.
Annual Convention, Washington, DC, November 2013. Munro, M. C., S. L. Huff, B. L. Marcolin, and D. R. Compeau.
Jarrahi, M. H. 2018. “Artificial Intelligence and the Future of 1997. “Understanding and Measuring User Competence.”
Work: Human-AI Symbiosis in Organizational Decision Information & Management 33 (1): 45–57.
Making.” Business Horizons 61 (4): 577–586. Neves, B. B., F. Amaro, and J. R. Fonseca. 2013. “Coming of
Kandlhofer, M., G. Steinbauer, S. Hirschmugl-Gaisch, and P. (Old) Age in the Digital age: ICT Usage and Non-Usage
Huber. 2016. “Artificial Intelligence and Computer Science Among Older Adults.” Sociological Research Online 18
in Education: From Kindergarten to University.” Paper (2): 22–35.
Presented at the 2016 IEEE Frontiers in Education Ng, W. 2012. “Can we Teach Digital Natives Digital Literacy?”
Conference (FIE). Computers & Education 59 (3): 1065–1078.
Katz, I. R. 2007. “Testing Information Literacy in Digital Noh, Y. 2017. “A Study on the Effect of Digital Literacy on
Environments: ETS’s ISkills Assessment.” Information Information Use Behavior.” Journal of Librarianship and
Technology and Libraries 26 (3): 3–12. Information Science 49 (1): 26–56.
Kim, Y. 2013. “A Study of Primary School Teachers’ Nomura, T., T. Kanda, and T. Suzuki. 2006. “Experimental
Awareness of Digital Textbooks and Their Acceptance of Investigation Into Influence of Negative Attitudes Toward
Digital Textbooks Based on the Technology Acceptance Robots on Human–Robot Interaction.” Ai & Society 20
Model.” Journal of Digital Convergence 11 (2): 9–18. (2): 138–150. doi:10.1007/s00146-005-0012-7.
14 B. WANG ET AL.

Norman, D. 2013. The Design of Everyday Things: Revised and Stembert, N., and M. Harbers. 2019. “Accounting for the
Expanded Edition. Basic books. Human When Designing with AI: Challenges Identified.”
Park, Y. J. 2013. “Digital Literacy and Privacy Behavior CHI’19-Extended Abstracts, Glasgow, Scotland Uk—May
Online.” Communication Research 40 (2): 215–236. 04-09, 2019.
Pett, M. A., N. R. Lackey, and J. J. Sullivan. 2003. Making Sense Su, G. 2018. “Unemployment in the AI Age.” AI Matters 3 (4):
of Factor Analysis: The Use of Factor Analysis for Instrument 35–43.
Development in Health Care Research. Sage. Tao, J., and T. Tan. 2005. “Affective Computing: A Review.”
Pettit, F. A. 2002. “A Comparison of World-Wide Web and Paper presented at the International Conference on
Paper-and-Pencil Personality Questionnaires.” Behavior Affective Computing and Intelligent Interaction.
Research Methods, Instruments, & Computers 34 (1): 50–54. Tarafdar, M., C. M. Beath, and J. W. Ross. 2019. “Using AI to
Poria, S., E. Cambria, R. Bajpai, and A. Hussain. 2017. “A Enhance Business Operations.” MIT Sloan Management
Review of Affective Computing: From Unimodal Analysis Review 60 (4): 37–44.
to Multimodal Fusion.” Information Fusion 37: 98–125. Tobin, C. D. 1983. “Developing Computer Literacy.”
Porter, C. E., and N. Donthu. 2006. “Using the Technology Arithmetic Teacher 30 (6): 22–60.
Acceptance Model to Explain how Attitudes Determine Vossen, S., J. Ham, and C. Midden. 2010. “What Makes Social
Internet Usage: The Role of Perceived Access Barriers Feedback from a Robot Work? Disentangling the Effect of
and Demographics.” Journal of Business Research 59 (9): Speech, Physical Appearance and Evaluation.” Paper pre-
999–1007. sented at the International Conference on Persuasive
Prior, D. D., J. Mazanov, D. Meacheam, G. Heaslip, and J. Technology.
Hanson. 2016. “Attitude, Digital Literacy and Self Weisberg, M. 2011. “Student Attitudes and Behaviors
Efficacy: Flow-on Effects for Online Learning Behavior.” Towards Digital Textbooks.” Publishing Research
The Internet and Higher Education 29: 91–97. Quarterly 27 (2): 188–196.
Rosling, A., and K. Littlemore. 2011. “Improving Student Wilson, H. J., and P. R. Daugherty. 2018. “Collaborative
Mental Models in a New University Information Setting.” Intelligence: Humans and AI Are Joining Forces.”
In Digitisation Perspectives, 89–101. Cham, Switzerland: Harvard Business Review.
Springer. Wilson, M., K. Scalise, and P. Gochyyev. 2015. “Rethinking
Russell, S., and P. Norvig. 2002. Artificial Intelligence: A ICT Literacy: From Computer Skills to Social Network
Modern Approach. Settings.” Thinking Skills and Creativity 18: 65–80.
Schriesheim, C. A., K. J. Powers, T. A. Scandura, C. C. Xiao, W., and M. Bie. 2019. “The Reform and Practice of
Gardiner, and M. J. Lankau. 1993. “Improving Construct Educational Technology Major in the Age of Artificial
Measurement in Management Research: Comments and a Intelligence 2.0.” Paper presented at the IOP Conference
Quantitative Approach for Assessing the Theoretical Series: Materials Science and Engineering.
Content Adequacy of Paper-and-Pencil Survey-Type Zhou, H., M. Itoh, and S. Kitazaki. 2021. “How Does
Instruments.” Journal of Management 19 (2): 385–417. Explanation-Based Knowledge Influence Driver Take-
Smith, R. G., and J. Eckroth. 2017. “Building AI Applications: Over in Conditional Driving Automation?” IEEE
Yesterday, Today, and Tomorrow.” AI Magazine 38 (1): 6–22. Transactions on Human-Machine Systems.