Computers in Human Behavior: Artificial Humans 1 (2023) 100014

Contents lists available at ScienceDirect

Computers in Human Behavior: Artificial Humans

journal homepage: www.journals.elsevier.com/computers-in-human-behavior-artificial-humans

MAILS - Meta AI literacy scale: Development and testing of an AI literacy

questionnaire based on well-founded competency models and
psychological change- and meta-competencies
Astrid Carolus a, 1, Martin J. Koch a, *, 1, Samantha Straka b, Marc Erich Latoschik b,
Carolin Wienrich c
a
University of Würzburg, Media Psychology, Sanderring 2, Würzburg, 97070, Bavaria, Germany
b
University of Würzburg, Human-Computer Interaction, Sanderring 2, Würzburg, 97070, Bavaria, Germany
c
University of Würzburg, Psychology of Intelligent Interactive Systems, Sanderring 2, Würzburg, 97070, Bavaria, Germany

A R T I C L E I N F O A B S T R A C T

Keywords: Valid measurement of AI literacy is important for the selection of personnel, identification of shortages in skill
AI literacy and knowledge, and evaluation of AI literacy interventions. A questionnaire is missing that is deeply grounded in
Questionnaire development the existing literature on AI literacy, is modularly applicable depending on the goals, and includes further
Psychological competencies
psychological competencies in addition to the typical facets of AIL. This paper presents the development and
Self-efficacy
validation of a questionnaire considering the desiderata described above. We derived items to represent different
Competence modeling
facets of AI literacy and psychological competencies, such as problem-solving, learning, and emotion regulation
in regard to AI. We collected data from 300 German-speaking adults to confirm the factorial structure. The result
is the Meta AI Literacy Scale (MAILS) for AI literacy with the facets Use & apply AI, Understand AI, Detect AI, and
AI Ethics and the ability to Create AI as a separate construct, and AI Self-efficacy in learning and problem-solving
and AI Self-management (i.e., AI persuasion literacy and emotion regulation). This study contributes to the
research on AI literacy by providing a measurement instrument relying on profound competency models. Psy­
chological competencies are included particularly important in the context of pervasive change through AI
systems.

1. Introduction AI patents is 76.6% between 2015 and 2021 (Zhang, Maslej, et al.,
2022). The broad spectrum of AI brings with it challenges for under­
It is an undeniable fact that Artificial Intelligence (AI) is coming into standing AI, as the underlying systems or capabilities of AI are complex
our daily lives. Interaction with AI or AI systems will become increas­ and challenging to grasp (Wienrich & Latoschik, 2021). To find one’s
ingly common for work or entertainment. Worldwide, about one-third of way in an AI-influenced world and to be able to act in a self-determined
all companies used AI in 2022 which is an increase by four points manner and participate in future developments, not only experts but
compared to 2021.42% have not yet started using AI but explored the also average users need an understanding of what AI is, what it can do,
topic of AI in 2022 (IBM, 2022). The demand for individuals skilled in AI and how they can benefit (Carolus et al., 2022; Wienrich, Carolus,
has steadily increased since 2014 compared to 2022, as the AI Adoption Markus, & Augustin, 2022). Just as computer skills became more
Index 2023 shows for North American, European, and other Western important a few years ago, AI skills are becoming more relevant today.
countries. For example, the number of AI-related job postings has This set of skills includes using, applying, or interacting with AI and is
increased on average from just above 0.50% in 2014 to 2.05% in 2022 in commonly referred to as ”AI literacy” (Long & Magerko, 2020). In­
the United States (Maslej et al., 2023). In parallel, more AI-based tech­ dividuals with high AI literacy will likely flourish in a working envi­
nologies are being developed, as the average annual growth rate of filed ronment rich with AI, while individuals with low AI literacy will have

* Corresponding author.
E-mail addresses: [email protected] (A. Carolus), [email protected] (M.J. Koch), [email protected] (S. Straka),
[email protected] (M.E. Latoschik), [email protected] (C. Wienrich).
1
Both authors contributed equally to this research.

https://doi.org/10.1016/j.chbah.2023.100014
Received 30 May 2023; Received in revised form 11 September 2023; Accepted 16 September 2023
Available online 5 October 2023
2949-8821/© 2023 The Authors. Published by Elsevier Inc. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-
nc-nd/4.0/).
A. Carolus et al. Computers in Human Behavior: Artificial Humans 1 (2023) 100014

problems when required to interact with AI. As automation and ”a bundle of four core capabilities”, namely technology-related, work-­
collaboration with AI will occur in many jobs (Frey & Osborne, 2017), related, human-machine-related, and learning-related capabilities. They
an individual’s current level of AI literacy might predict if they can argue that technological capabilities will be necessary to understand and
adapt to new technologies and if implementing AI-reliant workflows will use AI, as it is based on technology. However, they limit these capabil­
be successful. It is essential to develop suitable means to measure the AI ities to the use of tools and data literacy and do not include in-depth
literacy of individuals who are required to work with AI. programming skills to be part of AI literacy. Zhang, Lee, et al. (2022)
A reliable and valid measurement instrument for AI literacy is developed a curriculum for middle schools to foster AI literacy with
essential for selecting suitable personnel, identifying shortages in skills three components: AI concepts includes factual knowledge about AI and
and knowledge that can be addressed, and evaluating interventions that its concepts and technical details. Ethical and societal implications consists
focus on improving AI literacy. Specific tests and questionnaires were of the ability to understand the consequences of using AI for society, and
developed to measure AI literacy in educational settings where efforts AI Career Future concerns the impact of AI on future careers. However,
are made to implement AI literacy into school curricula and develop in-depth technical knowledge about the creation of AI is not included.
educational approaches for increasing AI literacy (Dai et al., 2020; For their measurement instrument on AI literacy, Wang and colleagues
Kandlhofer et al., 2016; Rodríguez-García, Moreno-León, Román-­ define AI literacy to have the components awareness, usage, evaluation,
González, & Robles, 2021; Wan et al., 2020a; Williams et al., 2019). and ethics but do not include the ability to develop AI applications as
Besides, there are few ways to measure AI literacy, and many scales are part of their conceptualization (Wang et al., 2022).
bound to specific contexts (e.g., useable only in an educational or Bloom’s taxonomy of educational objectives (Bloom, 1956) is also
medical setting) or need to be validated (Karaca et al., 2021; Wienrich & highly relevant to the definition of AI literacy and to the development of
Carolus, 2021). Moreover, no measurement instrument takes into ac­ AI literacy scales. The taxonomy includes different levels of educational
count psychological meta-competencies (Wienrich, Carolus, Markus, & goals, such as the ability to remember terms and concepts (i.e.,
Augustin, 2022). However, these are particularly important in work and Remember) or more complex operations, like the ability to analyze
adult education since the introduction of AI systems is often accompa­ complex matters. Even though it was not developed to function as a
nied by general change processes that must be mastered constructively. guideline for the construction of literacy conceptualizations or mea­
The goal of the present paper is to provide a measurement instrument surement scales, as Ng et al. (2021) conclude, most conceptualizations of
that deals with the desiderata of current instruments and is modular in AI literacy parallel Bloom’s taxonomy regarding their general configu­
addition. In the context of this article, a modular measurement instru­ ration of skills (Fig. 1). Since this taxonomy is the basis for numerous
ment is understood to be an instrument that consists of various com­ competence formulations in schools and universities, the present paper
ponents that can be used separately from one another. The present paper also relates to it by considering it a foundation for many AI literacy
addresses these research gaps by presenting an empirical study on the conceptualizations and scales.
systematic development and factorial validation of an AI literacy scale There is one central point in which we differ from Bloom’s taxonomy
that meets psychometric requirements, is cross-contextual applicable, is in our understanding of AI literacy. We do not expect all components of
embedded in the current literature on AI literacy, and considers psy­ AI literacy to be ordered in a strict hierarchical sense. Instead, we as­
chological meta-competencies. sume that they are loosely connected. For example, it is possible to be
able to create and develop AI without being able to make ethical con­
2. Theoretical background siderations and evaluate the advantages and disadvantages of the use of
AI.
Originally, ”literacy” referred to the basic knowledge to read and
write. More modern definitions apply a more general understanding of 2.1. Measuring AI literacy
literacy as the ”ability to identify, understand, interpret, create,
communicate and compute, using printed and written materials asso­ Several scales have been developed to measure AI literacy so far. As
ciated with varying contexts” (for Statistics, nd). This, thus, involves not many articles about AI literacy stem from an educational context, many
only basic skills of reading and writing but also more complex thought measurement instruments were developed for the evaluation of a spe­
processes of comprehension, interpretation, and creation. In recent cific intervention. Often, teaching success is measured with single-
years, the term literacy has been used for a broader array of compe­ choice or open-ended knowledge tests (e.g., Ali et al., 2019;
tencies regarding other domains (e.g., finance, health, or science). Most Kandlhofer et al., 2016; Ng et al., 2022; Rodríguez-García et al., 2021;
subtypes of literacy, however, focus on information technology (e.g., Wan et al., 2020b; Williams et al., 2019; Zhang, Lee, et al., 2022). The
digital literacy, media literacy, information literacy, technology literacy, advantage of these tests is the apparently higher quality of measure­
information technology literacy, social media literacy (Polanco-Levicán ment, which is only given to a limited extent with open answers, which
& Salvo-Garrido, 2022), digital interaction literacy (Carolus et al., are subject to personal opinions. An additional disadvantage of these
2022)). tests is that they often remain close to the content of the intervention to
AI literacy definitions differ in the exact number and configuration of be evaluated or to the content of the lesson. Other researchers in an
competencies. There are many different conceptualizations and defini­ educational context resort to self-assessments (e.g., Chai et al., 2020,
tions of AI literacy: Ng and colleagues, in their review on AI literacy 2021; Dai et al., 2020; Kim et al., 2021; Kim & Lee, 2022), which are
conceptualizations in education, postulate that these can be organized easier to carry out and more objective as no interpretation of answers is
into four concepts: Know & understand AI, Use & apply AI, Evaluate & necessary. Some research tends to use both options in combination
create AI, and AI ethics (Ng et al., 2021). They assume that AI literacy is (Kandlhofer et al., 2016; Zhang, Lee, et al., 2022). What all instruments
given if an individual knows the basic functions of AI and can use AI used in schools have in common is that their factorial structure was not
applications, can apply AI knowledge in different scenarios, can eval­ examined in large samples. Most of these questionnaires and tests might
uate, appraise, predict, and design AI applications, and can make ethical be helpful in evaluating specific interventions. However, they are less
considerations. Most conceptualizations of AI literacy consider users as suitable for the measurement of AI literacy in a broader spectrum of use
AI literate even if they do not have the in-depth technical knowledge and cases for two reasons: First, they heavily depend on the specific
cannot develop or create AI. For example, Long & Magerko define AI knowledge of the tested intervention. For example, in order to assess AI
literacy as a ”[…] set of competencies that enable individuals to criti­ literacy in general in work contexts, it should be possible to query
cally evaluate AI technologies; communicate and collaborate effectively general criteria, which can then be combined with context-specific as­
with AI; and use AI as a tool online, at home, and in the workplace” pects in a modular way. Second, a large proportion of these studies do
(Long & Magerko, 2020). Cetindamar et al. (2022) define AI literacy as not differentiate between different aspects of AI literacy. Especially for

2
A. Carolus et al. Computers in Human Behavior: Artificial Humans 1 (2023) 100014

Fig. 1. Bloom’s Taxonomy and AI literacy adapted from Ng et al. (2021).

Fig. 2. AI used at work (n = 178) as reported by the participants (N = 300).

Fig. 3. Conceptual model for the confirmatory factor analysis.

its extensive use in science and practice, it is important to differentiate students’ readiness for using artificial intelligence in their work.
between distinct facets of AI literacy so that the questionnaire can be Exploratory and confirmatory factor analysis showed a good fit for the
used economically and purposefully. 27-item scale, which tends to measure AI readiness in the four domains
At the time of finalizing this paper, there were three published scales of ”Cognition”, ”Ability”, ”Vision”, and ”Ethics”. Another study on
that could be used for a more general measurement of AI literacy and developing and validating a scale to measure AI literacy was published
one collection of items to measure AI literacy. Karaca and colleagues last year (Wang et al., 2022). The ”artificial intelligence literacy scale”
created a scale to specifically measure the AI Readiness of medical stu­ measures AI literacy with 12 items on the four dimensions of ”Aware­
dents in healthcare through a self-report scale (Karaca et al., 2021). ness”, ”Usage”, ”Evaluation”, and ”Ethics”. Again, the identified di­
However, adapting the scale to different professional fields would be mensions were confirmed by factor analysis. However, Wang and
easily possible. The Medical Artificial Intelligence Readiness Scale for colleagues draw heavily on existing literature on digital literacy to
Medical Students (MAIRS-MS) was developed to measure medical conceptualize AI literacy and develop their questionnaire. Both

3
A. Carolus et al. Computers in Human Behavior: Artificial Humans 1 (2023) 100014

Fig. 4. Structural equation model of the modified confirmatory factor analysis. The items are omitted.

questionnaires do not consider the current theoretical advancements behavior (called perceived control or self-efficacy) are central to the
regarding the conceptualization of AI literacy (Ng et al., 2021). The third intention of showing or changing a behavior. The theory of planned
instrument was recently presented by Pinski and Benlian (2023). It is behavior (Ajzen, 1985) postulates that intentions and perceived
designed to measure the AI literacy of non-experts in the working behavioral control are most important to predict human voluntary
context. Even though they refer to AI literacy conceptualizations (Long behavior (such as the use of AI or managing AI-induced changes). In­
& Magerko, 2020; Ng et al., 2021), their conceptualization of AI literacy tentions, in turn, are centrally predicted by attitudes towards the
and the structure of the questionnaire are not based on these concep­ behavior, the subjective norm (i.e., influences from the social environ­
tualizations. Instead, they follow conceptualizations from the field of ment), and perceived behavioral control. Thus, perceived behavioral
information systems (Schuetz et al., 2020) to guide their scale con­ control takes a unique role in that it influences behavior directly and
struction. Pinski and Benlian (2023) differentiate between explicit indirectly (i.e., via intentions). Even though the theory of planned
knowledge and tacit knowledge, which can be understood - and is behavior is used mainly to predict change in health behavior and has
measured in their instrument - as an experience rather than knowledge been criticized (Sniehotta et al., 2014), recently, it has successfully been
(i.e., ”I have experience in interaction with different types of AI […].”). applied to predict the intention to use AI in different domains such as
They also assume a ”socio-technical” perspective in the development of agriculture (Mohr & Kühl, 2021), and human resources (Alam et al.,
their measurement instrument. However, this does not refer to similar­ 2020), the intention to learn AI (Chai et al., 2021) or other related new
ities between human-AI interactions and human-human interaction. technologies (Zaman et al., 2021). Outside the theory of the planned
Instead, it is concerned with the knowledge of other human actors in the behavior framework, it was also shown that the subjective assessment of
field of AI. one’s competencies is central to the intention to use AI (Kwak et al.,
Laupichler et al. (2023) presented a collection of 38 items to measure 2022; Latikka et al., 2019). Consequently, according to an essential
AI literacy. They generated an initial set of items and asked experts in general psychological theory of intentional behavior, it is vital to mea­
the field of AI education to refine the items following the Delphi method. sure perceived behavioral control for the target domain of AI usage in
Their item collection targets non-expert users of AI. Items are only addition to other constructs, such as the attitude towards the usage of AI
loosely based on a recent AI literacy framework (Long & Magerko, or social influences. From a psychological point of view, it is, thus,
2020), which was used as an ”implicit decision support tool”. Also, no reasonable to resort to a self-report questionnaire to measure AI literacy.
factor analysis was conducted to test the factorial structure and advance However, the behavioral process does not end with the one-time
AI literacy conceptualizations. formation of a behavioral intention (Gollwitzer, 1990). In the further
The numerous current publications show that AI literacy is an course of the use of AI, the control of action and emotion processes is
important topic that is researched in very different and specialized ways. necessary for the successful long-term and sustainable use of AI (Goll­
This results in measuring instruments that, on their own, produce a witzer, 1990). According to Bandura (1997), several sources are central
coherent structure but ignore essential aspects and thus have only a very to the formation of perceived behavioral control, which he calls
limited scope of application. Established competence taxonomies like self-efficacy. Individuals experience higher self-efficacy when they are
those of Bloom (1956) and Ng et al. (2021) are not consistently used as a successful themselves, when they see others being successful, when they
theoretical basis for item formulation. Thus, the interpretation of latent experience no negative and high positive emotional arousal, and when
factors also remains arbitrary. Therefore, we base our measurement they are supported by others. Especially new developments in the field
instrument on the established competence taxonomy of Bloom (1956) of AI might hinder the long-term use of AI as they can have a negative
and AI literacy conceptualization of Ng et al. (2021). effect on perceived behavioral control: Innovations might unsettle po­
tential users, lead to failed behavior, and thus reduce perceived
behavioral control. In addition, negative emotional states of arousal
2.2. A psychological perspective on the measurement of AI literacy
might occur, which can have further adverse effects on perceived
behavioral control. We suggest that other psychological competencies
When measuring human competencies such as AI literacy, several
are central to mitigating adverse effects on perceived behavioral control
options (e.g., tests, observations of behavior, questionnaires) exist.
and, thus, the behavioral intention and actual use of AI. Learning,
Although tests tend to be more objective and observations tend to
problem-solving, and emotion regulation might be needed to mitigate
possess a higher validity, self-report questionnaires have a different
the adverse effects of innovations on perceived behavioral control.
advantage not despite but because they measure the self-perceived
Especially learning and problem-solving regarding AI can enable the
competence of an individual: According to the theory of planned
potential AI user to keep up with current developments in AI. Learning
behavior (Ajzen, 1985) or similar theories (e.g., adaptions to UTAUT2,
has been considered an essential part of or addition to AI literacy before
Venkatesh et al., 2012; Alalwan et al., 2015; Lallmahomed et al., 2017;
(Carolus et al., 2022; Cetindamar et al., 2022; Dai et al., 2020). These
Gao et al., 2015), the perceived capabilities regarding a particular

4
A. Carolus et al. Computers in Human Behavior: Artificial Humans 1 (2023) 100014

competencies presumably lead to higher future use of AI by reducing 3. Empirical study

failures leading to reduced perceived behavioral control. Additionally,
emotion regulation helps to reduce harmful and increase positive Empirical data were collected online from 300 individuals with a
emotional arousal (Carolus et al., 2022) and, by that, also increases first language of German on the 3rd and 4th of November 2022 using the
perceived behavioral control. We argue that an instrument for survey tool SoSci-Survey (Leiner, 2022). Participants were recruited
measuring AI literacy to predict and prepare AI use in a professional using Prolific. co, an online platform that provides contacts to potential
context should, from a psychological point of view, primarily focus on participants for online studies (Damer & Bradley, 2014). Participants
the subjective assessment of one’s competencies (i.e., behavioral control were eligible for the study if they spoke German. We made no further
or self-efficacy). From the perspective of the theory of planned behavior requirements regarding socio-demographic data or prior experience
(Ajzen, 1985), behavioral control takes on a central role. In addition to with or knowledge of AI. The participants received short information
subjective competence, other competencies are also critical, especially about the purpose of the study before they could decide whether they
to help predict and ensure the long-term use of AI. These include, in wanted to start the study or not. They received compensation worth
particular, the ability to learn, problem-solving skills, and emotion €3.38 for completion after their data was reviewed. 300 participants
regulation to compensate for failures and resulting negative emotional completed the study, two were rejected due to failed attention checks,
arousal. In addition, we also consider the ability to recognize and pre­ 13 participants decided to return their submission, and three partici­
vent the influences of human-like voice-based AI systems (Carolus et al., pants did not complete the study. Preliminary testing resulted in an
2022; Wienrich, Carolus, Roth-Isigkeit, & Hotho, 2022) as necessary. In average completion time of approximately 20 min. The average
the following, these competencies will be summarized under the term AI completion time was 16:05 min (SD = 5:46), amounting to an average
Self-management. We argue that AI Self-management is important to reward of approximately €12.8/hour. All participants were asked to give
ensure the prolonged and sustainable use of AI and, thus, consider it to informed consent prior to participation. In addition to the items we
be an essential part of our measurement instrument. generated for AI literacy following the conceptualization by Ng and
colleagues (Ng et al., 2021), the creation and development component of
2.3. Summary and present work (Bloom, 1956) and AI Self-management, anthropomorphism tendency,
attitudes towards AI, and the willingness to use technology and de­
In summary, different conceptualizations of AI literacy exist where mographic information were assessed mainly using standardized
the main focus is on the domains of basic knowledge about AI, the use of instruments.
AI, and ethical aspects of AI usage (Long & Magerko, 2020; Ng et al.,
2021). Several measurement instruments already exist. However, some 3.1. Sample
desiderata in the research on AI literacy measurement still need to be
worked on. In the present sample, the average age was 32.13 years (SD = 11.66
years, ranging from 18 to 72 years). Most participants lived in Germany
1. In general, few measurement instruments with initial validations and (77.00%) or Austria (7.00%). 145 participants considered themselves
theoretical foundations exist (Karaca et al., 2021; Wang et al., 2022), female (48.33%), while 152 participants identified as male (50.67%). 3
besides practically-oriented instruments often developed for a spe­ participants identified as diverse (1.00%). Participants were asked to
cific intervention study in an educational context. indicate their experience with AI by rating three statements ”I use
2. Most instruments are not based on established theoretical compe­ artificial intelligence at work”, ”I use artificial intelligence at school/
tency modeling, making interpreting latent factors seem arbitrary. university”, and ”I use artificial intelligence in my everyday life” on an
3. Fewer researchers include the development of AI as a part of AI lit­ 11-point Likert scale (0 = ”never or only very rarely” to 10 = ”very
eracy or allude to initial psychological components, as described often”). Almost one-fifth of the participants (19.67%) reported never
above (Carolus et al., 2022; Carolus & Wienrich, 2021; Wienrich, using AI in their everyday lives, at school/university, or at work. The
Carolus, Markus, & Augustin, 2022). average scores (and standard deviation) were M = 1.84 (2.79) for work,
4. The assessment of related psychological constructs we call AI Self- M = 1.21 (2.31) for school/university, and M = 3.73 (3.03) for everyday
management (i.e., emotion regulation, problem-solving, and life respectively showing rather low use of AI on the scale of 0 = ”never
learning in regard to AI) we deem vital information to support the or only very rarely” to 10 = ”very often”. Participants were also asked to
long-term and sustainable use of AI at the workplace and beyond and report the AI they use at work (Fig. 2). In total, the participants reported
has not been included in efforts to measure AI literacy. the use of 158 AI-based systems. The scope ranged from the use of de­
vices with AI, to the use of programs that include AI functions, to the
With these desiderata in mind, this paper aims to meaningfully autonomous implementation of machine learning processes. Ten AI
extend previous work on the conceptualization and measurement of AI frameworks and ten AI providers and environments (e.g., Amazon,
literacy by developing a measurement instrument that: Watson) were reported.

1. Is deeply grounded in the existing literature on AI literacy, 3.2. Measures

2. Is modular (i.e., including different facets that can be used inde­
pendently of each other) to be flexibly applicable in professional life All measures were administered online via SoSci-Survey in German.
depending on the goals, Prior to participation, the participants were informed about the general
3. Meets psychological psychometric requirements, purpose of the study and gave their informed consent.
4. And includes further psychological competencies in addition to the
classical facets of AI literacy. 3.2.1. AI literacy and AI self-management
After reviewing the literature on AI literacy described in the theo­
The present study contributes to the research on the conceptualiza­ retical background, we generated 56 items for the self-assessments in
tion and measurement of AI literacy by (a) presenting a newly developed different domains of AI literacy. We focused on the four superordinate
measurement instrument and (b) testing the factorial structure. The domains described by Ng et al. Ng et al. (2021) 1, namely Know & un­
measurement instrument differs from already existing instruments in derstand AI, Use & apply AI, Evaluate & create AI, and AI ethics. The
several essential ways. item construction was heavily guided by the conceptualization of Ng
et al. as each item was developed to directly represent one of their do­
mains of AI literacy. 15 Items were developed for Know & understand

5
A. Carolus et al. Computers in Human Behavior: Artificial Humans 1 (2023) 100014

AI, while 12 items were created for Use & apply AI. The original item (Rosseel, 2012) (version 0.6–12) and used robust Satorra-Bentler esti­
pool we generated for Evaluate & create AI consisted of 15 items for mations. In the first step, we included all items on AI literacy and AI
Evaluate and 9 for Create AI. Lastly, 5 items were created for AI ethics. In self-management. Based on the conceptual derivation/structure of the
addition to the items aiming at the assessment of the domains of AI items (3), we tested if the items loaded on the factor they were devel­
literacy, we generated 12 items (three per construct) to measure addi­ oped in reference to. The lower level factors Know & understand AI, Use
tional constructs derived from the literature as presented above we & apply AI, Evaluate & create AI, and AI ethics were expected to load on
deemed necessary for individuals working on and with AI. These con­ the second-level factor called AI literacy. The factors AI Problem solving,
structs include (a) the ability to manage one’s own emotions while AI Learning, AI Persuasion literacy, and AI Emotion regulation were
interacting with AI, called Emotion regulation, (b) the ability to recog­ expected to load on the second-level factor we called AI
nize if one’s decisions are influenced by AI and to stop this influence (i.e. Self-management; Fig. 3).
”AI persuasion literacy”), (c) the ability to solve problems encountered Because of an insufficient model fit, in the second step, we made the
while working with AI (i.e., AI problem solving), and (d) the ability to following changes: We removed 34 items that showed low factor load­
keep up to date with current developments and inform oneself about ings. In order to guarantee that the questionnaire still covers all the
new AI applications, called Learning. These abilities have in common domains of AI literacy, we ensured that only items were removed that
that they describe self-management aspects: They include managing doubled in content with the remaining items. Three items from the
one’s own emotions and decisions as well as the management of factor Know & understand AI were moved to a separate factor named
problem-solving and learning processes. In the first step, for each of the Detect AI. The model was changed so that the level one factor, Create AI,
domains, items were generated by one researcher. Then, the items were does not load on the level two factor AI literacy. Lastly, the level two
discussed, rephrased, rejected, and finalized by our team of researchers factor AI Self-management was split into the factors AI Self-efficacy
from the areas of human-computer interaction and psychology. The 12 (including AI Learning and AI Problem solving) and AI Self-
items on AI self-management and 56 items on AI literacy were admin­ competency (including AI Persuasion literacy and AI Emotion regula­
istered first. Each item included a statement about a specific ability tion. We re-run the confirmatory factor analysis with the changes. The
related to one of the domains of AI literacy or AI self-management (e.g., final questionnaire can be seen in the appendix. The model fit for the
”I can develop new AI applications.”). The participants were asked to modified model was good. Although the χ 2-test became significant
rate their own abilities using an 11-point Likert scale (0–10). We decided (χ 2(513) = 886.87, p < 0.001), the other model fit indices showed a
to use this scale because it can easily be understood as the certainty of good model fit (CFI = 0.926, RMSEA = 0.057, 95 %-CI [0.051, 0.063],
being able to show a behavior Bandura et al. (2006). There is no scale SRMR = 0.079). CFI > 0.9 and RMSEA < 0.08 indicate an acceptable
labeling to achieve an approximate metric scale level. The only addi­ model fit, while an SRMR > indicates no good fit (Kline, 2015).
tional information the participants receive is that a value of 0 means that All items loaded significantly on their respective factor (all p <
the ability is hardly or not at all pronounced, whereas a value of 10 0.001), and all level one factors loaded significantly on their respective
means that the ability is very well or almost perfectly pronounced. level two factor (all p < 0.001). All level two factors were significantly
correlated (all p < 0.01). Interestingly, AI Self-efficacy and AI Self-
3.2.2. Attitude towards AI competency showed very high correlations with AI literacy, whereas
To measure attitude toward AI, we used our own German translation they were still highly correlated with each other, but to a lesser extent.
of the General Attitude towards Artificial Intelligence Scale (GAAIS) The result of the confirmatory factor analysis is a measurement model
(Schepman & Rodway, 2020). The scale consists of 20 items that mea­ for 34 manifest items on AI literacy and related psychological compe­
sure positive (e.g., ”There are many beneficial applications of Artificial tencies (Fig. 4). A total of 18 items load on the level-one factors loading
Intelligence”) and negative (e.g., ”I think Artificial Intelligence is on the level-two factor AI literacy: Six items each load on the latent
dangerous”) attitudes towards AI. Participants rate their attitude to­ dimensions Use & Apply AI (Cronbach’s α = 0.93) and Know & Un­
wards AI on a 5-point Likert scale with the anchors strongly ”dis­ derstand AI (Cronbach’s α = 0.87). Three items each load on the latent
agree”/”somewhat disagree”/”neutral”/”somewhat agree”/”strongly dimensions Detect AI (Cronbach’s α = 0.77) and AI Ethics (Cronbach’s α
agree”. In our sample, both subscales showed high internal consistencies = 75). Four items directly load on the second-level latent dimension
(α = 0.88 for positive and α = 0.82 for negative attitude). Create AI (Cronbach’s α = 0.92). Three items each load on the first-level
latent dimensions AI Problem solving (Cronbach’s α = 0.84) and
3.2.3. Willingness to use technology Learning (Cronbach’s α = 0.84) loading on the second-level latent
The short scale for willingness to use technology (Neyer et al., 2012) dimension AI Self-Efficacy. For the second-level latent dimension AI
measures the acceptance of technology (i.e., ”I am curious about new Self-competency, three items load on the first-level latent dimensions of
technical developments.”), the competence (i.e.,” I usually find using AI Persuasion literacy (Cronbach’s α = 0.66) and AI Emotion regulation
modern technology to be a challenge.”) and the perceived control (i.e.,” (Cronbach’s α = 0.71). All second-level latent dimensions were signifi­
Whether or not I succeed in using modern technology largely depends on cantly correlated with each other (Fig. 4).
me.”) when using new technologies with four items per scale. Partici­ In the third step, we additionally included the subscales for attitude
pants rate the items regarding their willingness to work with technology towards AI and willingness to use technology as additional latent vari­
on a 5-point Likert scale (from 1 = ”not true at all” to 5 = ”completely ables to the structural equation model. All items loaded significantly on
true”). The items for ”competence” were recoded so that a high value
indicates a high perceived competence in line with the other two scales.
All three scales showed good internal consistencies (all α > 0.81). We are Table 1
aware that the use of 5-point Likert scales is often discouraged (e.g., Descriptive statistics for AI attitudes (positive and negative) and willingness to
Dawes, 2008; van Beuningen et al., 2014). Although being aware of the use new technology (acceptance, competence, control).
disadvantages of 5-point Likert scales we decided to stick as close to the mean sd median min max
original scales as possible. Descriptive information for the attitude to­ AI attitude Positive 3.60 0.57 3.67 1.50 4.92
wards AI and willingness to use technology can be seen in Table 1. attitude
Negative 2.74 0.67 2.75 1.13 4.50
attitude
3.3. Results
Willingness to use Acceptance 3.56 0.94 3.75 1.00 5.00
To test the factorial structure of our measurement instrument, we technology Competence 4.83 0.82 4.29 1.00 5.00
Control 3.85 0.68 4.00 1.75 5.00
calculated a confirmatory factor analysis with the package lavaan for R

6
A. Carolus et al. Computers in Human Behavior: Artificial Humans 1 (2023) 100014

their respective scale (all p < .001). The correlations of the latent vari­ the creation of AI as a part (or dimension) of AI literacy. Instead, our
ables can be seen in Table 2. The model showed an acceptable model fit research suggests that Create AI should be operationalized as a separate
(χ 2(2035) = 3004.35, p < 0.001, CFI = 0.900, RMSEA = 0.043, 95 %-CI skill that is related to, but not an inherent part of, AI literacy. This is in
[0.039, 0.047], SRMR = 0.069). A CFI > 0.9 and an RMSEA < 0.08 line with most conceptualizations of AI literacy where the development
indicate an acceptable fit, while an SRMR > 0.05 indicates no good fit of AI is not explicitly mentioned (Dai et al., 2020; Kong et al., 2021; Long
(Kline, 2015). & Magerko, 2020), however, is in conflict with the conceptualization by
Self-assessed competence in the use of new technologies correlated Ng et al. (2021). The structure found, where Create AI does not load on
positively only with Create AI. It did not correlate with AI Self-efficacy AI literacy but is correlated to it, covers this dichotomy well, and the
and only negatively with AI literacy and AI Self-competency. All AI dimension Create AI can be measured modularly along with AI literacy.
competencies correlated positively with the perceived control over new Concerning the domain AI literacy, we found further discrepancies
technology and acceptance of new technology except Create AI, which between the first specified model and the final model. Interestingly,
did not correlate with perceived control over new technology. A positive items for the evaluation of AI did not load on one factor with items on
attitude towards AI was positively correlated with all AI competencies. A Create AI but loaded on the factor for Know & understand AI. It seems
negative attitude was negatively correlated to AI literacy and AI Self- that the ability to evaluate AI is more closely related to the general
competency but was not correlated to Create AI and AI Self-efficacy. knowledge and understanding of AI than to the ability to develop AI.
Presumably, according to the subjects’ self-assessment, for the evalua­
4. Discussion tion of AI systems, precise knowledge and understanding (i.e., Know &
understand AI) are more important than the ability to actually develop
The aim of this paper was to develop and validate a questionnaire to AI. This fits in with the general picture that Create AI is separate (Dai
assess AI literacy that targets several desiderata of current measurement et al., 2020; Kong et al., 2021; Long & Magerko, 2020) and no part of AI
instruments. In contrast to existing questionnaires, first, the question­ literacy. Our findings, thus, partly contradict the conceptualization by
naire should be deeply grounded in the existing literature on AI literacy. Ng et al. (2021), who included Create AI as a component of their AI
Second, the questionnaire should be modular (i.e., including different literacy conceptualization. Additionally, we found a common factor for
facets that can be used independently of each other) to be flexibly the ability to detect AI similar to the dimensions ”Awareness” (e.g., ”I
applicable in professional life, depending on the goals and use cases. can distinguish between smart devices and non-smart devices.”, Wang
Third, the questionnaire should meet psychological requirements, and, et al. (2022)). Possibly, recognizing AI does not seem to be necessarily
fourth, it should include further psychological competencies in addition tied to knowing and understanding AI, but also is an independent
to the classical facets of AI literacy. For this purpose, the data of 300 competence. This is in line with the conceptualization by Long and
German-speaking adults were analyzed with confirmatory factor ana­ Magerko (2020), who included the ability to recognize AI in their
lyses. The analyses resulted in a questionnaire consisting of 34 items to framework. However, as expected, we found the other factors with the
measure AI literacy and psychological competencies necessary for the expected items to load on the second-level latent dimension called AI
sustainable use of AI literacy. Items were generated in direct reference to literacy.
established conceptualizations of competencies (Bloom, 1956) and AI In place of one domain called AI Self-management, which includes
literacy (Ng et al., 2021). The measurement instrument can be used to the factors AI Problem solving, AI Learning, AI Persuasion literacy, and
measure AI literacy and additional psychological competencies inde­ AI Emotion regulation, we found 2 s-level latent domains we called AI
pendent of the context. The MAILS will have an impact on different Self-efficacy (including the factors AI Problem solving and AI Learning)
fields as its application can help select suitable personnel, identify and AI Self-competency (including AI Persuasion literacy and AI
shortages in skills and knowledge that can be addressed, and evaluate Emotion regulation). Possibly, problem-solving and learning are com­
interventions. petencies that are primarily aimed at managing information and infor­
Instead of the eight factors (four derived from the literature, Ng et al., mation processing, while persuasion literacy and emotion regulation
2021) and four to measure specific psychological aspects we deemed also focus on the management of information, albeit with greater per­
essential) that loaded on 2 s-level factors (i.e., AI literacy and AI sonal value (own decisions and emotions). We follow the opinion that
Self-management), we found the facets Use & Apply AI, Know & un­ these factors are important for the prolonged and sustainable use of AI
derstand AI, Detect AI, and AI Ethics which loaded on a second-level tools (Carolus et al., 2022; Carolus & Wienrich, 2021; Cetindamar et al.,
factor called AI literacy. Nearly all scales of the questionnaire showed 2022; Dai et al., 2020; Wienrich, Carolus, Markus, & Augustin, 2022). A
acceptable (Cronbach’s α > .7), good (Cronbach’s α > 0.8), or high clear correlation emerged between AI Self-efficacy and AI
(Cronbach’s α > 0.9) values for Cronbach’s alpha supporting the com­ Self-competency, although the subordinate factors do not load on one
posite reliability of each scale. Only the latent dimensions of AI common factor. It is possible that similar cognitive processes are
Persuasion literacy showed a slightly low value (Cronbach’s α = 0.66). necessary for the management of processes that are more concerned
Interestingly, the facet Create AI did not load on AI literacy and was only with the processing of information (AI Self-efficacy) or emotions and
correlated to it with a r = 0.5. This result from our measurement model decisions (AI Self-competency), leading to a high correlation among
can be seen as support for the conceptualizations which do not include both domains. However, enough uniqueness exists in both constructs. So

Table 2
Correlations from the structural equation model.
2. 3. 4. 5. 6. 7. 8. 9.

1. AIL 0.49*** 0.86*** 0.93*** 0.14* 0.20** 0.47*** − 0.17* 0.47***

2. Create 0.24** 0.63*** − 0.18* − 0.02 0.21*** 0.10 0.14*
3. AISC 0.72*** 0.24** 0.31*** 0.36*** − 0.27** 0.28**
4. AISE 0.06 0.17** 0.48*** − 0.10 0.33***
5. Competence 0.41*** 0.42*** − 0.39*** 0.22**
6. Control 0.46*** − 0.27*** 0.36***
7. Acceptance − 0.30*** 0.00
8. Negative − 0.43***
9. Positive

Note. * indicates p < .05, ** indicates p < .01, *** indicates p < .001; AIL = AI literacy, AISC = AI self-competency, AISE = AI self-efficacy.

7
A. Carolus et al. Computers in Human Behavior: Artificial Humans 1 (2023) 100014

far, conceptual thinking existed to include such aspects (Carolus et al., exploring available options to test the predictive validity and construct
2022; Carolus & Wienrich, 2021; Cetindamar et al., 2022; Dai et al., validity of the scale in samples of university students. For this purpose, it
2020; Wienrich, Carolus, Markus, & Augustin, 2022), Wienrich, Carolus, seems possible to ask students in AI and machine learning university
Roth-Isigkeit, & Hotho, 2022ut no measurement tool yet. Our mea­ courses to complete the scale and also report their grades in the
surement instrument, therefore, provides new value and an essential respective course. Additionally, a comparison with the other existing
contribution to existing considerations. This is the first step regarding instruments regarding their predictive validity (e.g., use them to predict
the individual differences in such a general measuring instrument the quality of future AI-related behavior) would be interesting to iden­
(Wienrich, Carolus, Roth-Isigkeit, & Hotho, 2022). tify the worth of the additional scales we included. Also, testing the
There are high correlations between the constructs of our question­ discriminant validity of the scale by comparing it with other constructs
naire (i.e., AI literacy, AI Self-competency, Create AI, and AI Self- (e.g., intelligence, IT literacy, data literacy) might be highly relevant to
efficacy), but still enough differentiation that they can be understood prove the instrument’s worth. A translation of the MAILS seems
as unique constructs. All second-level latent dimensions positively reasonable to measure AI literacy in different linguistic and cultural
correlate with positive attitude toward AI. AI literacy and AI Self- contexts. Additional effort is necessary to translate the questionnaire
competency negatively correlate to a negative attitude towards AI, and validate it. Ensuring measurement invariance would be an impor­
while Create AI and AI Self-efficacy do not. Presumably, individuals tant condition for using the MAILS cross-linguistically and culturally.
with a higher positive attitude also deal with the topic more often and
are, therefore, more competent. Similar is true for negative attitude (but
4.2. Conclusion
only for some of the constructs), with lower negative attitude going hand
in hand with higher literacy and self-competence. Possibly, individuals
The current study aimed to develop and validate a questionnaire to
with lower negative attitude (i.e., less fear or anxiety) show higher
measure AI literacy and include psychological competencies that might
values of AI literacy and AI Self-competency because they interact with
be helpful in predicting the prolonged and sustainable use of AI. We
AI more frequently. Alternatively, individuals with higher self-
based our developed items on the existing literature on AI literacy and
competency are better at regulating their emotions and attitudes (i.e.,
psychological competencies. Overall, we found the factors Use & Apply
less fear or anxiety), leading to less negative attitude. In a similar way,
AI, Know & understand AI, and AI Ethics (Ng et al., 2021) with the
the three dimensions of the willingness to use technology are positively
addition of Detect AI. The factor Evaluate & Create AI was not found.
related to most of the dimensions of our questionnaire. These correlative
While the items on the evaluation of AI loaded on Know & understand
findings suggest that there are relations between competencies related
AI, the items on the creation of AI formed their own factor that cannot be
to AI and attitudes towards AI as well as the willingness to use new
seen as an inherent part of AI literacy. Instead of finding one superor­
technology. Individuals with higher positive attitudes, lower negative
dinate factor for the psychological competencies related to AI, we found
attitudes, and more willingness to use technology are more likely to
two (i.e., AI Self-efficacy and AI Self-competency). We mainly found
consider themselves competent (Ajzen, 1985; Wienrich & Carolus, 2021;
positive relations for our questionnaires’ dimensions with attitudes to­
Wienrich, Carolus, Roth-Isigkeit, & Hotho, 2022). One reason may be
ward AI and willingness to use technology. Create AI is a notable special
that these individuals are also more likely to use AI and thus become
case in that its correlations to attitudes and willingness are compara­
more competent.
tively low compared to the other dimensions of our scale. The developed
scale will contribute to the current research on AI literacy and will
4.1. Limitations and future work
facilitate the implementation of AI into working environments by
providing a valid measurement for AI literacy and related psychological
Several limitations need to be mentioned in regard to the empirical
competencies that can be used by various practitioners and researchers.
study presented in this paper. Our sample was collected online and is
In addition, it has theoretical implications as it helps to get a better
specific to German-speaking individuals who live in Germany or Austria.
understanding of AI literacy. Further research will be needed to relate
Also, it was not possible to use an already existing and validated in­
our measure to other valid measures of AI literacy and compare their
strument on AI literacy to validate our questionnaire, as would be the
predictive validity. The current study contributes to the existing
gold standard for scale development. Although other instruments exist,
research by providing a measurement instrument for AI literacy that is
none have been validated with external criteria so far. Even though we
based on the current literature on AI literacy, includes important psy­
chose confirmatory factor analysis to test our models, we made changes
chological constructs, and has a valid factorial structure. The instrument
to the model, thus, it is necessary to consider our approach exploratory.
will be helpful for researchers, practitioners, and educators who plan to
It is, therefore, highly important to confirm the factorial structure we
measure AI literacy and related constructs.
identified in an independent sample.
Future research should aim to test the factorial structure we identi­
fied in independent samples. The most critical next step, however, is the Declaration of competing interest
validation of our questionnaire. This could be done by either correlating
our questionnaire results with results of tests, tasks, or the evaluations of The authors have no competing interest to declare.
individual AI literacy by an expert or by correlating it to a validated
questionnaire that might be published in the future. Alternatively, other Acknowledgements
external criteria could be used: We could test if the instrument is capable
of detecting expected group differences or finding change after in­ This work was supported by the German Federal Ministry of Labour
terventions/in the course of professional studies. We are currently and Social Affairs [DKI.00.00030.21].

Appendix A. Structure and all items of the Meta AI literacy Scale and sources that motivate each facet

8
A. Carolus et al. Computers in Human Behavior: Artificial Humans 1 (2023) 100014

AI literacy

Apply AI (Ng et al., 2022)

1. I can operate AI applications in everyday life.
2. I can use AI applications to make my everyday life easier.
3. I can use artificial intelligence meaningfully to achieve my everyday goals.
4. In everyday life, I can interact with AI in a way that makes my tasks easier.
5. In everyday life, I can work together gainfully with an artificial intelligence.
6. I can communicate gainfully with artificial intelligence in everyday life.
Understand AI (Ng et al., 2022)
7. I know the most important concepts of the topic ”artificial intelligence”.
8. I know definitions of artificial intelligence.
9. I can assess what the limitations and opportunities of using an AI are.
10. I can assess what advantages and disadvantages the use of an artificial intelligence entails.
11. I can think of new uses for AI.
12. I can imagine possible future uses of AI.
Detect AI (Long & Magerko, 2020; Wang et al., 2022)
13. I can tell ifI am dealing with an application based on artificial intelligence.
14. I can distinguish devices that use AI from devices that do not.
15. I can distinguish if I interact with an AI or a ”real human”.
AI Ethics (Ng et al., 2022)
16. I can weigh the consequences of using AI for society.
17. I can incorporate ethical considerations when deciding whether to use data provided by an AI.
18. I can analyze AI-based applications for their ethical implications.
Create AI (Ng et al., 2022)
19. I can design new AI applications.
20. I can program new applications in the field of ”artificial intelligence”.
21. I can develop new AI applications.
22. I can select useful tools (e.g., frameworks, programming languages) to program an AI.
AI Self-Efficacy
AI Problem solving (Ajzen, 1985)
23. I can rely on my skills in difficult situations when using AI.
24. I can handle most problems in dealing with artificial intelligence well on my own.
25. I can also usually solve strenuous and complicated tasks when working with artificial intelligence well.
Learning (Carolus et al., 2022; Cetindamar et al., 2022; Dai et al., 2020)
26. I can keep up with the latest innovations in AI applications.
27. Despite the rapid changes in the field of artificial intelligence, I can always keep up to date.
28. Although there are often new AI applications, I manage to always be ”up-to date”.
AI Self-Competency
AI Persuasion literacy (Carolus et al., 2022)
29. I don’t let AI influence me in my everyday decisions.
30. I can prevent an AI from influencing me in my everyday decisions.
31. I realise if artificial intelligence is influencing me in my everyday decisions.
AI Emotion regulation (Carolus et al., 2022)
32. I keep control over feelings like frustration and anxiety while doing everyday things with AI.
33. I can handle it when everyday interactions with AI frustrate or frighten me.
34. I can control my euphoria that arises when I use artificial intelligence for everyday purposes.

References Cetindamar, D., Kitto, K., Wu, M., Zhang, Y., Abedin, B., & Knight, S. (2022). Explicating
ai literacy of employees at digital workplaces. Transactions on Engineering
Management, Early Access, 1–14.
Ajzen, I. (1985). From intentions to actions: A theory of planned behavior. Action control,
Chai, C. S., Lin, P.-Y., Jong, M. S.-Y., Dai, Y., Chiu, T. K., & Qin, J. (2021). Perceptions of
11–39 (Springer, Berlin).
and behavioral intentions towards learning artificial intelligence in primary school
Alalwan, A. A., Dwivedi, Y. K., Rana, N. P., Lal, B., & Williams, M. D. (2015). Consumer
students. Educational Technology & Society, 24(3), 89–101.
adoption of internet banking in Jordan: Examining the role of hedonic motivation,
Chai, C. S., Wang, X., & Xu, C. (2020). An extended theory of planned behavior for the
habit, self-efficacy and trust. Journal of Financial Services Marketing, 20(2), 145–157.
modelling of Chinese secondary school students’ intention to learn artificial
Alam, M. S., Khan, T.-U.-Z., Dhar, S. S., & Munira, K. S. (2020). HR professionals’
intelligence. Mathematics, 8(11), 2089.
intention to adopt and use of artificial intelligence in recruiting talents. Business
Dai, Y., Chai, C. S., Lin, P. Y., Jong, M. S. Y., Guo, Y., & Qin, J. (2020). Promoting
Perspective Review, 2(2), 15–30.
students’well-being by developing their readiness for the artificial intelligence age.
Ali, S., Payne, B. H., Williams, R., Park, H. W., & Breazeal, C. (2019). Constructionism,
Sustainability, 12(16), 1–15.
ethics, and creativity: Developing primary and middle school artificial intelligence
Damer, E., & Bradley, P. (2014). Prolific.
education. In International workshop on education in artificial intelligence k-12
Dawes, J. (2008). Do data characteristics change according to the number of scale points
(eduai’19) (pp. 1–4). Palo Alto, California: mit media lab.
used? An experiment using 5-point, 7-point, and 10-point scales. International
Bandura, A. (1997). Self-Efficacy: The Exercise of control. New York: WH Freeman/Times
Journal of Market Research, 50(1), 61–104.
Books/Henry Holt & Co.
Frey, C. B., & Osborne, M. A. (2017). The future of employment: How susceptible are jobs
Bandura, A., et al. (2006). Guide for constructing self-efficacy scales. Self-efficacy beliefs
to computerisation? Technological Forecasting and Social Change, 114, 254–280.
of adolescents, 5(1), 307–337.
Gao, S., Yang, Y., & Krogstie, J. (2015). The adoption of smartphones among older adults
van Beuningen, J., van der Houven, K., & Moonen, L. (2014). Measuring well-being: An
in China. In International conference on informatics and semiotics in organisations (pp.
analysis of different response scales. The Hague: Statistics Netherlands.
112–122). Toulouse: Springer. IFIP.
Bloom, B. S. (1956). Taxonomy of educational objectives, Handbook I: The cognitive domain.
Gollwitzer, P. M. (1990). Action phases and mind-sets. Handbook of motivation and
New York: David McKay Co Inc.
cognition: Foundations of social behavior, 2, 53–92.
Carolus, A., Augustin, Y., Markus, A., & Wienrich, C. (2022). Digital interaction literacy
IBM. (2022). Ibm global ai adoption index 2022. IBM. Technical report.
model – conceptualizing competencies for literate interactions with voice-based ai
Kandlhofer, M., Steinbauer, G., Hirschmugl-Gaisch, S., & Huber, P. (2016). Artificial
systems. Computers and Education: Artificial Intelligence, 4, Article 100114.
intelligence and computer science in education: From kindergarten to university. In
Carolus, A., & Wienrich, C. (2021). Towards a holistic approach and measurement of
2016 IEEE frontiers in education conference (FIE) (pp. 1–9). Pennsylvania: Eire. IEEE.
humans interacting with speech-based technology. In A. Carolus, I. Siebert, &
Karaca, O., Çalışkan, S. A., & Demir, K. (2021). Medical artificial intelligence readiness
C. Wienrich (Eds.), 1st AI-DEbate workshop : Workshop establishing an InterDisciplinary
scale for medical students (mairs-ms)–development, validity and reliability study.
pErspective on speech-BAsed TEchnology (pp. 1–42). Magdeburg, Germany: Otto von
BMC Medical Education, 21(1), 1–9.
Guericke University Magdeburg.

9
A. Carolus et al. Computers in Human Behavior: Artificial Humans 1 (2023) 100014

Kim, S., Jang, Y., Kim, W., Choi, S., Jung, H., Kim, S., & Kim, H. (2021). Why and what to J. Sheard (Eds.), Proceedings of the 52nd ACM technical symposium on computer science
teach: Ai curriculum for elementary school. In Proceedings of the AAAI conference on education (pp. 177–183). New York, NY, USA: ACM.
artificial intelligence (pp. 15569–15576). Virtual event. AAAI. Rosseel, Y. (2012). lavaan: An R package for structural equation modeling. Journal of
Kim, S. W., & Lee, Y. (2022). The artificial intelligence literacy scale for middle school Statistical Software, 48(2), 1–36.
students. Journal of the Korea Society of Computer and Information, 27(3), 225–238. Schepman, A., & Rodway, P. (2020). Initial validation of the general attitudes towards
Kline, R. B. (2015). Principles and practice of structural equation modeling. New York: artificial intelligence scale. Computers in human behavior reports, 1, Article 100014.
Guilford Publications. Schuetz, S., & V, V. (2020). Research perspectives: The rise of human machines: How
Kong, S.-C., Cheung, W. M.-Y., & Zhang, G. (2021). Evaluation of an artificial intelligence cognitive computing systems challenge assumptions of user-system interaction.
literacy course for university students with diverse study backgrounds. Computers Journal of the Association for Information Systems, 21(2), 460–482.
and Education: Artificial Intelligence, 2, Article 100026. Sniehotta, F. F., Presseau, J., & Araújo-Soares, V. (2014). Time to retire the theory of
Kwak, Y., Ahn, J.-W., & Seo, Y. H. (2022). Influence of AI ethics awareness, attitude, planned behaviour. Health Psychology Review, 8(1), 1–7.
anxiety, and self-efficacy on nursing students’ behavioral intentions. BMC Nursing, for Statistics, U. I. (n.d.). .
21(1), 2–8. Venkatesh, V., Thong, J. Y., & Xu, X. (2012). Consumer acceptance and use of
Lallmahomed, M. Z., Lallmahomed, N., & Lallmahomed, G. M. (2017). Factors information technology: Extending the unified theory of acceptance and use of
influencing the adoption of e-government services in Mauritius. Telematics and technology. MIS Quarterly, 36(1), 157–178.
Informatics, 34(4), 57–72. Wang, B., Rau, P. L. P., & Yuan, T. (2022). Measuring user competence in using artificial
Latikka, R., Turja, T., & Oksanen, A. (2019). Self-efficacy and acceptance of robots. intelligence: Validity and reliability of artificial intelligence literacy scale. Behaviour &
Computers in Human Behavior, 93, 157–163. Information Technology. NULL(NULL):1–14.
Laupichler, M. C., Aster, A., & Raupach, T. (2023). Delphi study for the development and Wan, X., Zhou, X., Ye, Z., Mortensen, C. K., & Bai, Z. (2020a). Smileycluster: Supporting
preliminary validation of an item set for the assessment of non-experts’ ai literacy. accessible machine learning in k-12 scientific discovery. In Proceedings of the
Computers and Education: Artificial Intelligence, 4, Article 100126. interaction design and children conference (Vols. 23–35). New York, NY, USA: ACM.
Leiner, D. J. (2022). Sosci survey version 3.3.08. . Wan, X., Zhou, X., Ye, Z., Mortensen, C. K., & Bai, Z. (2020b). Smileycluster: Supporting
Long, D., & Magerko, B. (2020). What is ai literacy? Competencies and design accessible machine learning in k-12 scientific discovery. In Proceedings of the
considerations. In Conference on human factors in computing systems - proceedings (pp. interaction design and children conference (pp. 23–35). London: ACM.
1–16). Honolulu, Hawaii: ACM. Wienrich, C., & Carolus, A. (2021). Development of an instrument to measure
Maslej, N., Fattorini, L., Brynjolfsson, E., Etchemendy, J., Ligett, K., Lyons, T., conceptualizations and competencies about conversational agents on the example of
Manyika, J., Ngo, H., Niebles, J. C., Parli, V., Shoham, Y., Wald, R., Clark, J., & smart speakers. Frontiers of Computer Science, 3, 70.
Perrault, R. (2023). The ai index 2023 annual report. Technical report, AI Index Steering Wienrich, C., Carolus, A., Markus, A., & Augustin, Y. (2022). Ai literacy:
Committee. Institute for Human-Centered AI, Stanford University. Kompetenzdimensionen und einflussfaktoren im kontext von arbeit. Working Paper.
Mohr, S., & Kühl, R. (2021). Acceptance of artificial intelligence in German agriculture: Wienrich, C., Carolus, A., Roth-Isigkeit, D., & Hotho, A. (2022). Inhibitors and enablers to
An application of the technology acceptance model and the theory of planned explainable ai success: A systematic examination of explanation complexity and
behavior. Precision Agriculture, 22(6), 1816–1844. individual characteristics. Multimodal Technologies and Interaction, 6(12), 106.
Neyer, F. J., Felber, J., & Gebhardt, C. (2012). Development and validation of a brief Wienrich, C., & Latoschik, M. E. (2021). Extended artificial intelligence: New prospects of
measure of technology commitment. Diagnostica, 58(2), 87–99. human-ai interaction research. Frontiers in Virtual Reality, 2, 94.
Ng, D. T. K., Leung, J. K. L., Chu, S. K. W., & Qiao, M. S. (2021). Conceptualizing ai Williams, R., Park, H. W., & Breazeal, C. (2019). A is for artificial intelligence. In
literacy: An exploratory review. Computers and Education: Artificial Intelligence, 2, S. Brewster, G. Fitzpatrick, A. Cox, & V. Kostakos (Eds.), Proceedings of the 2019 CHI
Article 100041. conference on human factors in computing systems (pp. 1–11). New York, NY, USA:
Ng, D., Luo, W., Chan, H., & Chu, S. (2022). An examination on primary students’ ACM.
development in ai literacy through digital story writing. Computers & Education: Zaman, U., Zahid, H., Habibullah, M. S., & Din, B. H. (2021). Adoption of big data
Artificial Intelligence, 3, Article 100054. analytics (bda) technologies in disaster management: A decomposed theory of
Pinski, M., & Benlian, A. (2023). Ai literacy - towards measuring human competency in planned behavior (dtpb) approach. Cogent Business & Management, 8(1), 1–20.
artificial intelligence. In Proceedings of the 56th Hawaii international conference on Zhang, H., Lee, I., Ali, S., DiPaola, D., Cheng, Y., & Breazeal, C. (2022). Integrating ethics
system sciences (pp. 165–174). Maui, USA: IEEE Computer Society Press. and career futures with technical learning to promote ai literacy for middle school
Polanco-Levicán, K., & Salvo-Garrido, S. (2022). Digital interaction literacy model – students: An exploratory study. International Journal of Artificial Intelligence in
conceptualizing competencies for literate interactions. International Journal of Education, 1–35. NULL(NULL).
Environmental Research and Public Health, 19(14), 1–16. Zhang, D., Maslej, N., Brynjolfsson, E., Etchemendy, J., Lyons, T., Manyika, J., Ngo, H.,
Rodríguez-García, J. D., Moreno-León, J., Román-González, M., & Robles, G. (2021). Niebles, J. C., Sellitto, M., Sakhaee, E., Shoham, Y., Clark, J., & Perrault, R. (2022).
Evaluation of an online intervention to teach artificial intelligence with learningml The ai index 2022 annual report. Technical report, AI Index Steering Committee. Stanford
to 10-16-year-old students. In M. Sherriff, L. D. Merkle, P. Cutter, A. Monge, & Institute for Human-Centered AI, Stanford University.

10