GLOBAL PERSPECTIVES

AND INSIGHTS
Artificial Intelligence – Considerations for the
Profession of Internal Auditing
Special Edition
About The IIA Contents
The Institute of Internal Auditors (IIA) Introduction .............................................................................................. 2
is the internal audit profession’s most
widely recognized advocate,
Putting AI Into Context .............................................................................. 2
educator, and provider of standards, AI – The Basics ........................................................................................... 3
guidance, and certifications.
Established in 1941, The IIA today Big Data and Algorithms ..................................................................... 3
serves more than 190,000 members Types of AI .......................................................................................... 3
from more than 170 countries and
territories. The association’s global
AI Opportunities and Risks ........................................................................ 4
headquarters are in Lake Mary, Fla., Opportunities...................................................................................... 4
USA. For more information, visit
Risks .................................................................................................... 4
www.globaliia.org.
Internal Audit’s Role .................................................................................. 5
Reader Feedback AI Competencies: Filling the Understanding Gap ............................... 6
Send questions or comments to Reemphasizing Cyber Resilience ........................................................ 6
[email protected].
AI Auditing Framework.............................................................................. 6
Previous Issues Strategy ............................................................................................... 6
Previous issues of Global Perspectives Components........................................................................................ 7
and Insights on a variety of topics can Closing Thoughts ....................................................................................... 8
be found at www.theiia.org/gpi.

Disclaimer
The opinions expressed in Global
Perspectives and Insights are not
necessarily those of the individual
contributors or of the contributors’
employers.

Copyright
Copyright © 2017 by The Institute of
Internal Auditors, Inc. All rights
reserved.

globaliia.org 1
Artificial Intelligence
Considerations for the Profession of Internal Auditing
Introduction This application was demonstrated publicly to a wide
audience in 2011 when IBM’s AI platform Watson won a
Artificial intelligence (AI) is a broad term that refers to Jeopardy! exhibition on prime time TV. According to IBM
technologies that make machines “smart.” Organizations Research, IBM is “guided by the term ‘augmented
are investing in AI research and applications to intelligence’ rather than ‘artificial intelligence,’” and
automate, augment, or replicate human intelligence — focuses “on building practical AI applications that assist
human analytical and/or decision-making — and the people with well-defined tasks.” Human expertise
internal auditing profession must be prepared to fully develops technologies to make machines smart, and
participate in organizational AI initiatives. smart machines, in turn, augment human capabilities.

There are many other terms related to AI, such as, deep There is already widespread application of AI across
learning, machine learning, image recognition, natural- diverse sectors (publicly held, privately held,
language processing, cognitive computing, intelligence government, and nonprofit) and industries. Consider, for
amplification, cognitive augmentation, machine example, that AI enables a number of new and novel
augmented intelligence, and augmented intelligence. AI, capabilities that were impossible just a few years ago:
as used here, encompasses all of these concepts.
 Automobile manufacturers to develop self-driving
Putting AI Into Context vehicles.

AI is not new. According to the McKinsey Global  Online search engines to deliver targeted search
Institute’s (MGI) discussion paper “Artificial Intelligence: results.
The Next Digital Frontier,” the idea of AI dates back to
 Social media organizations to recognize faces in
1950 when Alan Turing first proposed that a machine photographs and filter newsfeeds.
could communicate well enough to convince a human
evaluator that it, too, was human.  Media companies to recommend books or shows to
subscribers.
While AI represents a series of significant
 Retailers to create customized online experiences for
advancements in technology, it was not the first, and shoppers.
likely will not be the last. Looking back over the last few
decades, the advent of computers, PCs, spreadsheets,  Logistics companies to route optimal paths for
deliveries.
relational databases, sophisticated connectivity, and
similar technological advancements have all impacted  Governments to predict epidemics.
how organizations operate and accomplish their
objectives. AI is poised to do the same with the
 Marketing professionals to deliver hyper-
personalized content to customers in real time.
potential to be as or more disruptive than many
previous technological advances.  Virtual assistants to use voice-controlled natural
language to interface with consumers.
AI can be viewed as the latest significant advancement
But it is not only new and novel activities affected by AI.
on a continuum of advancements that have occurred due
More mundane tasks that have been occurring for
to technology improvements. What is new is the
decades are being improved by AI such as loss modeling,
advancement and scalability of technologies that have
credit analysis, valuations, transaction processing, and a
unleashed the practical application of AI.
host of others.

globaliia.org 2
 Global Perspectives: Artificial Intelligence

It is critical that internal auditors pay attention to the

practical application of AI in business, and develop AUDIT FOCUS
competencies that will enable the internal auditing
IIA Standard 1210: Proficiency (Excerpt)
profession to provide AI-related advisory and assurance
services to organizations in all sectors and across all Internal auditors must possess the knowledge,
industries. skills, and other competencies needed to perform
their individual responsibilities. The internal audit
AI is dependent on big data and algorithms, and it can be
activity collectively must possess or obtain the
intimidating, especially for internal audit activities and
knowledge, skills, and other competencies needed
organizations that have yet to master big data. But
to perform its responsibilities.
internal auditors do not have to be data scientists or
quantitative analysts to understand what AI can do for 1210.A3 – Internal auditors must have sufficient
organizations, governments, and societies at large. knowledge of key information technology risks
and controls and available technology-based audit
This paper:
techniques to perform their assigned work.
However, not all internal auditors are expected to
 Presents an overview of AI basics.
have the expertise of an internal auditor whose
 Explores internal audit’s roles in AI. primary responsibility is information technology
 Discusses AI risks and opportunities. auditing.

 Introduces a framework for internal auditors (the

Framework).
while other data may be publicly available or purchased
This paper is Part I of a three-part series. Parts II and III from external sources.
will provide more detailed information on and practical
application of the Framework. To put big data to good use, organizations develop
algorithms. An algorithm is a set of rules for the machine
AI – The Basics to follow. An algorithm is what enables a machine to
quickly process vast amounts of data that a human
Big Data and Algorithms cannot reasonably process, or even comprehend. The
AI is powered by algorithms, and algorithms are fueled performance and accuracy of algorithms is very
by big data, so before an organization embarks on AI, it important. Algorithms are initially developed by humans,
should have a strong foundation in big data. And before so human error and biases (both intentional and
internal audit can think about addressing AI, it should unintentional) will impact the performance of the
already have a strong foundation in big data. For algorithm. Faulty algorithms can produce minor
comprehensive guidance on understanding and auditing undesirable glitches in an organization’s operations, or
big data, including a discussion of opportunities and risks, major catastrophic outcomes. It is generally recognized
and a sample work program, see The IIA’s “GTAG: that flawed algorithms, at least in part, fueled the 2008
Understanding and Auditing Big Data,” available free to global financial crisis.
IIA members and available to non-members through The
Types of AI
IIA Bookstore (www.theiia.org).
In The Conversation’s “Understanding the four types of
Big data means more than just large amounts of data — AI, from reactive robots to self-aware beings,” Arend
big data refers to data (information) that reaches such Hintze, assistant professor of Integrative Biology &
high volume, variety, velocity, and variability that Computer Science and Engineering at Michigan State
organizations invest in system architectures, tools, and University, outlines four types of AI:
practices specifically designed to handle the data. Much
of this data may be generated by the organization itself, Type I. Reactive machines: This is AI at its simplest.
Reactive machines respond to the same

globaliia.org 3
 Global Perspectives: Artificial Intelligence

situation in exactly the same way, every time.

AI Opportunities and Risks
An example of this is a machine that can beat
world-class chess players because it has been The first step toward understanding the organization’s AI
programmed to recognize the chess pieces, opportunities and risks is to thoroughly understand the
know how each moves, and can predict the organization’s big data opportunities and risks. Again, for
next move of both players. comprehensive guidance on understanding and auditing
big data, including a discussion of opportunities and risks,
Type II. Limited memory: Limited memory AI machines and a sample work program, see The IIA’s “GTAG:
can look to the past, but the memories are not Understanding and Auditing Big Data,” available free to
saved. Limited memory machines cannot build IIA members and available to non-members through The
memories or “learn” from past experiences. IIA Bookstore (www.theiia.org).
An example is a self-driving vehicle that can
decide to change lanes because a moment ago Examples of AI opportunities and risks include:
it noted an obstacle in its path.
Opportunities
Type III. Theory of mind: Theory of mind refers to the
idea that a machine could recognize that
 The ability to compress the data processing cycle.

others it interacts with have thoughts,  The ability to reduce errors by replacing human
feelings, and expectations. A machine actions with perfectly repeatable machine actions.
embedded with Type III AI would be able to  The ability to replace time-intensive activities with
understand others’ thoughts, feelings, and time-efficient activities (process automation),
expectations, and be able to adjust its own reducing labor time and costs.
behavior accordingly.
 The ability to have robots or drones replace humans
Type IV. Self-awareness: A machine embedded with in potentially dangerous situations.
Type IV AI would be self-aware. An extension  The ability to make better predictions, for everything
of “theory of mind,” a conscious or self-aware from predicting sales of certain goods in particular
machine would be aware of itself, know about markets to predicting epidemics and natural
its internal states, and be able to predict the catastrophes.
feelings of others.
 The ability to drive revenue and grow market share
In other words, a Type II self-driving vehicle would decide through AI initiatives.
to change lanes when a pedestrian is in its path, simply Risks
because it recognizes the pedestrian as an obstacle. A
Type III self-driving vehicle would understand that the  The risk that unidentified human biases will be
pedestrian would expect the vehicle to stop, and a Type imbedded in the AI technology.
IV self-driving vehicle would know that it should stop  The risk that human logic errors will be imbedded in
because that is what the self-driving vehicle would want the AI technology.
if it (the self-driving vehicle) were in the path of another
oncoming vehicle. Wow.
 The risk that inadequate testing and oversight of AI
results in ethically questionable results.
Most “smart machines” today are manifestations of Type  The risk that AI products and services will cause
I or Type II AI. Ongoing research and development harm, resulting in financial and/or reputational
initiatives will enable organizations to advance toward damage.
practical applications of Type III and Type IV AI.
 The risks that customers or other stakeholders will
not accept or adopt the organization’s AI initiatives.

globaliia.org 4
 Global Perspectives: Artificial Intelligence

 The risk that the organization will be left behind by

competitors if it does not invest in AI. AUDIT FOCUS
 The risk that investment in AI (infrastructure, IIA Standard 2120: Risk Management (Excerpt)
research and development, and talent acquisition)
The internal audit activity must evaluate the
will not yield an acceptable ROI.
effectiveness and contribute to the improvement
More in-depth information on AI risks will be presented of risk management processes.
in Parts II and III of this three-part Global Perspectives
and Insights series, which will provide recommendations 2120.A1 – The internal audit activity must
on leveraging the Framework to provide AI-related evaluate risk exposures relating to the
assurance and advisory services. organization’s governance, operations, and
information systems regarding the:
Internal Audit’s Role
 Achievement of the organization’s objectives.
Internal audit is adept at evaluating and understanding
the risks and opportunities related to the ability of an
 Reliability and integrity of financial and
operational information.
organization to meet its objectives. Leveraging this
experience, internal audit can help an organization  Effectiveness and efficiency of operations and
evaluate, understand, and communicate the degree to programs.
which artificial intelligence will have an effect (negative  Safeguarding of assets.
or positive) on the organization’s ability to create value in
the short, medium, or long term. Internal audit can  Compliance with laws, regulations, policies,
procedures, and contracts.
engage through at least five critical and distinct activities
related to artificial intelligence:

 For all organizations, internal audit should include AI  Internal audit should ensure the moral and ethical
issues that may surround the organization’s use of AI
in its risk assessment and consider whether to
are being addressed.
include AI in its risk-based audit plan.

 For organizations exploring AI, internal audit should  Like the use of any other major system, proper
governance structures need to be established and
be actively involved in AI projects from their
internal audit can provide assurance in this space.
beginnings, providing advice and insight contributing
to successful implementation. However, to avoid the Regardless of the specific activities performed, internal
perception of or actual impairments to both audit is well-suited to be a key contributor to an
independence and objectivity, internal audit should organization’s AI-related activities. Internal audit:
not own, nor be responsible for, the implementation
of AI processes, policies, or procedures.  Understands the strategic objectives of the
organization, and the processes implemented to
 For organizations that have implemented some achieve those objectives.
aspect of AI, either within its operations (such as a
manufacturer using robotics on a production line) or  Is able to evaluate whether AI activities are
accomplishing their objectives.
incorporated into a product or service (such as a
retailer customizing product offerings based on  Can provide internal assurance over management’s
purchase history), internal audit should provide risk management activities relevant to AI risks.
assurance on management of risks related to the  Is perceived as a trusted advisor that can positively
reliability of the underlying algorithms and the data support the adoption of AI to improve business
on which the algorithms are based. processes or enhance product and service offerings.

globaliia.org 5
 Global Perspectives: Artificial Intelligence

Internal auditing should approach AI as it approaches the board, and the audit committee. A good place to
everything — with systematic, disciplined methods to start is with The IIA’s thought leadership on AI, and The
evaluate and improve the effectiveness of risk IIA’s supplemental guidance on topics like big data and
management, control, and governance processes talent management.
related to AI.
Reemphasizing Cyber Resilience
AI Competencies: Filling the Understanding Gap Cybersecurity threats continue to define our times. The
The pool of talent for technology professionals with AI adoption and evolution of AI will force organizations to
expertise is reportedly small. Organizations who want to reemphasize their cyber resilience capabilities. As AI
participate in the AI revolution need to grow or acquire becomes more powerful and more decisions are handed
talent with competencies in a multitude of areas such as: off to new, complicated, and opaque algorithms, using
huge data sets, protecting these systems from outside,
 Natural language processing. malevolent forces is critical to success. A 2014 EY report
 Application program interfaces (APIs) such as facial defined cyber resilience as the ability to resist, react to,
recognition, image analytics, and text analytics. and recover from cyberattacks — and modify an
environment to increase security and sustainability over
 Algorithms and advanced modeling.
time. Cyber resiliency is critical for any organization
 Probabilities and applied statistics. relying increasingly on AI.
 Data analytics.
Among all the complexity surrounding cybersecurity,
 Software engineering. there are four key areas where internal audit can have an

 Programming language. immediate impact:

 Machine learning.
 Provide assurance over readiness and response to
 Computer vision. cyber threats.

 Robotics.  Communicate to executive management and the

board the level of risk to the organization and efforts
While a handful of organizations in the technology,
to address such risks.
automotive, manufacturing, financial services, and
utilities industries seem to be leading the AI  Work collaboratively with IT and other parties to
revolution, it is hard to imagine an organization that ensure effective defenses and responses are in place.
will not be impacted by AI. Just as computers,  Facilitate communication and coordination among
spreadsheets, and distributed processing were a focus all parties in the organization regarding the risk.
of select industries in their early stages, ultimately all The potentially disastrous effects of a cybersecurity
organizations adopted aspects of these technologies. breach involving AI cannot be overstated. If not already
As AI becomes more mainstream, it is hard to imagine in place, CAEs need to rapidly build cybersecurity within
any internal audit activity that will not need to be their teams.
ready to provide its organization with AI-related
assurance and advisory services.
AI Auditing Framework
How can CAEs upskill the internal audit activity to be The Framework is comprised of six components, all set
ready for the challenge? The first step is recognizing that within the context of an organization’s AI strategy.
new skillsets are required. Collectively, the internal audit
activity must have a sufficient understanding of AI, how Strategy
the organization is using it, and the risks that AI Each organization’s AI strategy will be unique based
represents to the organization. The CAE must be able to on its approach to capitalizing on the opportunities AI
communicate this understanding to senior management, provides. An organization’s AI strategy might be an

globaliia.org 6
 Global Perspectives: Artificial Intelligence

obvious extension of the organization’s overall digital  Information privacy and security throughout the
or big data strategy — organizations with a well- data lifecycle (data collection, use, storage, and
developed and implemented digital/big-data strategy destruction).
are one step ahead in AI. According to MGI,
organizations that “combine strong digital capability,  Roles and responsibilities for data ownership and
use throughout the data lifecycle.
robust AI adoption, and a proactive strategy see
outsize financial performance.” Data Quality
The completeness, accuracy, and reliability of the data on
Internal audit must consider an organization’s AI
which AI algorithms are built are critical. Unfortunately, it
strategy first. Does the organization have a defined
is not unusual for organizations to have a poorly defined,
strategy toward AI? Is it investing in AI research and
incoherent structure to their data. Often, systems do not
development? Does it have plans in place to identify
communicate with each other or do so through
and address AI threats and opportunities? AI can
complicated add-ons or customizations. How this data is
become a competitive advantage for organizations, and
brought together, synthesized, and validated is crucial.
internal audit should help management and the board
realize the importance of formulating a deliberate AI Measuring Performance of AI
strategy consistent with the organization’s objectives.
As organizations integrate AI into their activities,
performance metrics should be defined to tie AI activities
Components
to business objectives and clearly illustrate whether AI is
AI Governance effectively supporting the achievement of those
AI governance refers to the structures, processes, and objectives. Management must actively monitor the
procedures implemented to direct, manage, and performance of its AI activities.
monitor the AI activities of the organization in pursuit
of achieving the organization’s objectives. The level of The Human Factor
formality and structure for an organization’s AI Algorithms are developed by humans. Human error and
governance will vary based on the specific biases (both intentional and unintentional) will impact
characteristics of that organization. Regardless of the the performance of the algorithm. The human factor
specific approach, however, AI governance establishes component considers whether:
accountability and oversight, helps to ensure that those
responsible have the necessary skills and expertise to  The risk of unintended human biases factored into AI
effectively monitor AI, and helps to ensure the design is identified and managed.
organization’s values are reflected in its AI activities.  AI has been effectively tested to ensure that results
This last point should not be overlooked or given little reflect the original objective.
attention. AI activities must result in decisions and
actions that are in line with the ethical, social, and legal  AI technologies can be transparent given the
complexity involved.
responsibilities of the organization.
 AI output is being used legally, ethically, and
Data Architecture and Infrastructure responsibly.
AI data architecture and infrastructure will likely be one It is widely recognized that human error is the most
in the same as the organization’s architecture and common cause of information privacy and security
infrastructure for handling big data. It includes breaches. Similarly, the human factor component
considerations for: addresses the risk of human error compromising the
ability of AI to deliver the expected results.
 The way that data is accessible (metadata,
taxonomy, unique identifiers, and naming
conventions).

globaliia.org 7
 Global Perspectives: Artificial Intelligence

The Black Box Factor

According to the Merriam-Webster online dictionary, a AUDIT FOCUS
black box is “a usually complicated electronic device whose Key IIA Standards
internal mechanism is usually hidden from or mysterious to
the user; broadly: anything that has mysterious or The IIA’s International Standards for the
unknown internal functions or mechanisms.” Professional Practice of Internal Auditing includes
several standards that are particularly relevant to
As organizations advance to implementing Type III and AI, including:
Type IV AI technologies — utilizing machines or platforms
that can learn on their own or communicate with each  IIA Standard 1210: Proficiency
other — how the algorithms are operating becomes less
transparent or understandable. The black box factor will
 IIA Standard 2010: Planning

become more and more of a challenge as an  IIA Standard 2030: Resource Management
organization’s AI activities become more sophisticated.  IIA Standard 2100: Nature of Work

Closing Thoughts  IIA Standard 2110: Governance

 IIA Standard 2130: Control

The internal auditing profession cannot be left behind in
what may be the next digital frontier — artificial  IIA Standard 2200: Engagement Planning

intelligence. To prepare, internal auditors must  IIA Standard 2201: Planning Considerations
understand AI basics, the roles that internal audit can
and should play, and AI risks and opportunities. To meet
 IIA Standard 2210: Engagement Objectives

these challenges, internal auditors should leverage the  IIA Standard 2220: Engagement Scope

Framework to deliver systematic, disciplined methods to  IIA Standard 2230: Engagement Resource
evaluate and improve the effectiveness of risk Allocation
management, control, and governance processes related
to AI.
 IIA Standard 2240: Engagement Work
Program
This paper is Part I of a three-part series. Part II will  IIA Standard 2310: Identifying Information
provide more detailed information and
Complete text of the Standards is available at
recommendations regarding the AI Governance; Data
www.theiia.org. Each standard is complemented
Architecture and Infrastructure; and Data Quality
by an Implementation Guide.
components of the Framework. Part III will provide more
detailed information and recommendations regarding
the Measuring Performance, Human Factor, and the
Black Box Factor components of the Framework. Parts II
and III will include relevant engagement objectives and
procedures which internal audit activities can use to
customize an AI audit program to fit their organizations’
risk profile and strategic objectives.

globaliia.org 8