Decision Support Systems 30 Ž2001.

437–449
www.elsevier.comrlocaterdsw

Supporting managers’ internal control evaluations: an expert

system and experimental results
Chuleeporn Changchit a , Clyde W. Holsapple b,) , Donald L. Madden c
a
Management Sciences Department, UniÕersity of Iowa, Iowa City, IA, USA
b
School of Management, Carol M. Gatton College of Business Administration, UniÕersity of Kentucky, Lexington, KY 40506-0034, USA
c
School of Accountancy, UniÕersity of Kentucky, Lexington, KY, USA
Accepted 1 June 2000

Abstract

Internal control issues are of significance to entities to assure the accuracy, reliability, and timeliness of the financial
reports. Although management is responsible for maintaining an effective internal control system, the literature contains no
example of a system devised to aid managers detecting internal control weaknesses. This type of system might prove to be
highly beneficial. Nonetheless, it is not obvious that this system would yield positive results because managers may not be
willing or feel comfortable using it. This study reports on the development of such system, plus empirical testing of its value
and managers’ perceptions on its usefulness. q 2001 Elsevier Science B.V. All rights reserved.

Keywords: Expert system; Internal control; Auditing; Supporting manager

1. Introduction such decisions as they are being made. Thus, it

seems prudent for managers to approach decision
Evaluation and design of internal control systems making with the support of an internal control per-
is of crucial importance to management. Yet, it is a spective. Here, we report on the initial study of an
subject that receives too little attention in the man- expert system for facilitating the transfer of internal
agement literature. Managers tend not to be trained control knowledge to practicing managers, thereby
in the internal control arts, the detection of problems supporting their decisions from an internal control
tends to be left to auditors, and internal controls are perspective.
often viewed mainly as a means for preventing inten- Both accounting firms and researchers have de-
tional irregularities. However, managers’ decisions voted significant effort to the development of deci-
can have internal control implications, perhaps unin- sion aids, especially expert systems, to assist auditors
tentional or unrealized. Auditors do not oversee all in making the complex audit decisions encountered
in today’s business environment w7,9,15,17,27,29,31x.
Several expert systems had been built to assist audi-
)
Corresponding author. Tel.: q1-606-257-5236; fax: q1-606-
tors in evaluating internal control systems w5,10,12,
257-8031. 13,20,21,31,35x. These systems are generally devel-
E-mail address: [email protected] ŽC.W. Holsapple.. oped using either professional literature for a knowl-

0167-9236r01r$ - see front matter q 2001 Elsevier Science B.V. All rights reserved.
PII: S 0 1 6 7 - 9 2 3 6 Ž 0 0 . 0 0 1 2 7 - 5
438 C. Changchit et al.r Decision Support Systems 30 (2001) 437–449

edge source, or interview or protocol analysis to sisting managers evaluate existing internal control
acquire knowledge from human experts w23,28x. Such systems in their organizations. We describe a labora-
systems have been successfully applied and there is tory experiment designed to examine the usefulness
no need to examine the benefits of using them to of this expert system, including hypotheses to be
help auditors. However, there are no previously re- tested, independent and response variables, construc-
ported studies of expert systems devised to aid man- tion of experimental materials, subjects and their
agers in evaluating internal control weaknesses. Be- tasks, and statistical analysis procedures. The paper
cause it is an organization’s management, not audi- closes with highlights of research findings, notes
tors, who bear ultimate responsibility for maintaining about contributions of the research, and identification
an effective internal control, there is a need to study of directions for future related research.
the feasibility and value of equipping managers with
expert systems that give internal control advice.
Nonetheless, it is not obvious that the use of such
a system would yield positive results because man- 2. Background
agers typically have neither specific background in
internal control concepts nor hands-on experience Internal control is defined as a process designed
with expert systems. The managers may not be to provide reasonable assurance regarding the
willing to learn the concept of the internal controls achievement of objectives in the following cate-
or may not feel comfortable using the expert sys- gories: reliability of financial reporting, effectiveness
tems. For example, a manager may provide incorrect and efficiency of operations, and compliance with
inputs, not understand system outputs, find it diffi- applicable laws and regulation w1,2,25,32x. Maintain-
cult to work with an expert system, or resist consid- ing an effective internal control system is crucial to
ering the advice it gives. any business organization. Without the ability to
The main purpose of this research effort is to ensure the accuracy and reliability of accounting
address the question of whether an expert system can information, a business organization could not sur-
be devised to assist managers detect potential weak- vive in a competitive environment w5x. The problem
nesses of internal control systems in their organiza- of checking internal control weaknesses is a Anonde-
tions more efficiently andror more effectively. A terministic polynomialB problem w6,22x. Such prob-
subsequent experiment is also conducted to examine lems are often solved best by using the rules of
its value to managers. The extent of its assistance is thumb of experienced auditors. When such rules can
gauged in terms of the managers’ effectiveness, effi- be incorporated as reasoning knowledge in an expert
ciency, and satisfaction in using the system to evalu- system, the potential of developing an expert system
ate internal controls. If such a system can be shown to aid in solving audit problems is quite high.
to be both capable of development and beneficial, A wide variety of prototype and commercial sys-
then we will have established that the distribution of tems has been developed for assisting auditors evalu-
auditors’ expertise in evaluating internal control sys- ate their clients’ internal control systems w18,26,30x.
tems to a wide number of users can be accomplished Representative examples of prototype systems and
via expert system technology. The result is a new expert systems implemented for evaluating a client’s
way for increasing the efficiency and effectiveness internal control system are shown in Table 1
of ongoing decision-making processes concerned w5,12,13,20,21,31,35x. These systems aimed at help-
with internal control. Such expert systems could be ing auditors, not an organization’s management.
developed and deployed by either an organization or Some of the terms used in the systems are easy to
its auditors. understand by auditors, but not the novice users.
The following section discusses the literature on These systems are generally designed to help audi-
expert system applications for internal control evalu- tors determine the extent of other tests they will
ations. The next section provides the theoretical basis perform in conducting an audit. If an auditor deter-
for designing the experiment. We then outline the mines that the client’s internal controls are designed
construction of an expert system developed for as- properly and are functioning as designed, he or she
 C. Changchit et al.r Decision Support Systems 30 (2001) 437–449 439

Table 1 Ž1. Technological Acceptance Model ŽTAM. and Ž2.

Systems for internal control evaluations Induced Value Approach. Several hypotheses stem
Name DeveloperŽs. from the TAM. The experimental methodology in-
APE: Audit Planning Denna corporates the Induced Value Approach. An overview
and Evidence of these theories is presented in this section.
Inherent Risk Analysis KPMG Peat Marwick
PLANET Price Waterhouse
3.1. Technology acceptance model (TAM)
IRE Dhar et al.; Peters
Risk Advisor Coopers & Lybrand
ANSWER Arthur Young The TAM provides a basis for explaining the
CCRr36 Advisor Ernst & Young determinants of computer acceptance and user be-
CFILE KPMG Peat Marwick havior w14x. The model suggests that computer usage
C&L Control Risk Assessor Coopers & Lybrand is determined by two criteria, which are perceived
ICE Kelly
usefulness and perceived ease of use. The Aperceived
ICES Grudnitski & Arthur Young
Internal Control Analyzer Gal usefulnessB is defined as Athe prospective users’
Internal Control Expert Deloitte & Touche subjective probability that using a specific applica-
Flow Eval Ernst & Young tion system will increase his or her job performance
Systematic Price Waterhouse within an organizational contextB. The perceived
TICOM Bailey et al.
ease of use is defined as the degree to which the
prospective user expects the target system to be free
of effort w14x.
can reduce direct testing of account balances accord-
3.2. Induced Õalue approach
ingly w19x.
Noteworthy is that the professional literature The induced value approach was used in this
makes it clear that the responsibility for adopting experiment to induce optimal behavior from partici-
sound accounting policies, maintaining an adequate pants w34x. This approach emphasizes the importance
internal control system, and making fair representa- of a reward mechanism to overcome possible subjec-
tions in the financial statements rests with manage- tive tendencies and to focus on participation. It notes
ment rather than with the auditors w3,11x. However, that if well-designed laboratory experiments could
the literature does not report on expert systems that mirror actual organizations in key aspects, then labo-
can assist managers evaluate internal control systems ratory experimentation could play a very useful role.
of their companies. There are several reasons that The key underlying concept in the induced value
such systems deserve to be investigated. First, estab- approach is that a subject’s utility is an increasing
lishment and supervision of internal control systems function of the monetary reward. This implies that,
are the responsibility of management, not auditors. given everything else is equal, the participants will
Second, compared to an auditor, an organization’s try to maximize their reward.
manager should possess more knowledge about the
company and thus have greater immediate insight
into the operation of its internal control systems. And 4. The construction of an expert system for inter-
third, the evaluation of internal control systems can nal control evaluations
be done more often if it is performed by the manage-
ment because auditors are usually expensive and are The expert system developed in this research
not available upon request. offers advice about weaknesses found in evaluating
internal control systems in the sales and collection
cycle of medium-size merchandising organizations.
3. Theoretical basis The prospective users are managers who are not
experienced experts in auditing. The knowledge in-
The experimental methodology and hypotheses in corporated into the system represents primarily the
this dissertation research are based on two theories: expertise of one expert auditor who is a partner in a
440 C. Changchit et al.r Decision Support Systems 30 (2001) 437–449

major international accounting firm. The expert has 4.2. Rule set specification
more than 10 years of experience in the area of
internal control evaluation and demonstrated signifi- Knowledge acquired from the expert was repre-
cant interest in the research project. The tool selected sented in sets of rules. A couple of rule examples are
for developing the expert system was an integrated paraphrased in Table 2.
artificial intelligence environment called GURU By processing sets of such rules, the system infers
w8,24x. The version 3.01 of GURU was used to a recommendation of potential internal control weak-
implement this expert system. Construction of the nesses. This recommendation identifies significant
expert system is outlined below. internal control weaknesses discovered in the situa-
tion being evaluated and indicates resulting expo-
sures that could occur in a user’s organization.
4.1. Knowledge acquisition
4.3. Testing
Knowledge for the expert system was acquired
via a 6-month series of interviews with the expert. Once a prototype of the expert system had been
The expert was asked to identify all potential weak- developed, the expert was asked to test the expertise
nesses that may occur in the sales and collection captured in the rule sets in order to ensure the
cycle of a medium-size merchandising organization. validity of the system. Validation of the system is
The result was a list of 126 internal control weak- often considered the cornerstone of expert system
nesses. The expert was asked further to describe, in evaluation w4x. It is the process of analyzing the
detail, the techniques and processes he used to dis- knowledge and decision-making capabilities of the
cover each of these weaknesses in a client’s internal expert system w30x. Test cases were developed and
control system. The reasons for each decision-mak- used to examine whether the expert system did offer
ing heuristic were also acquired in an attempt to sufficiently good and timely advice compared to the
develop an expert system that would be able to human expert.
emulate both the expert’s knowledge and his reason- The test cases were generated from the manipula-
ing behavior. tion of several cues for detecting the potential weak-

Table 2
Examples of rule sets
Rules Remarks
R1
v
PERFORM ASK1. Ž1. ASK1 asks a user to enter the name of the personŽs. who
v
If the function of receiving cash or cheque and the receives cash or cheque and the name of the personŽs. who
function of recording cash or cheque are performed records cash or cheque.
by the same person, perform ADVICE1. Ž2. ADVICE1 presents the recommendation of an internal.
Ž3. Control weakness found in the organization and the reason
why it is considered a weakness.
R2
v
PERFORM ASK2. Ž1. ASK2 asks a user if the personŽs. who receives cash
v
If the function of receiving cash or cheque and the or cheque and the personŽs. who records cash or cheque
function of recording cash or cheque are performed by have a family relationship.
persons who have a family relationship, perform ASK3. Ž2. ASK3 asks a user to enter the relationship of the
v
If such relationship is considered critical under personŽs. who receives cash or cheque and the personŽs.
the internal control concept, perform ADVICE2. who records cash or cheque.
Ž3. ADVICE2 presents the recommendation of an
internal control weakness found in the organization
and the reason why it is considered a weakness.
 C. Changchit et al.r Decision Support Systems 30 (2001) 437–449 441

nesses in an internal control system over the sales internal control evaluation with no decision aid vs.
and collection cycle. These cues were obtained from internal control evaluation with support of the expert
a review of auditing texts, accounting texts, and system. Three response variables were measured as
input from accounting professors and experienced follows.
auditors. To prevent any bias in designing the case
studies, the expert who participated in developing the 5.1.1. Accuracy score
prototype expert system was not allowed to partici- Accuracy of decision-making is examined as a
pate in generating case studies. measure of the system’s effectiveness w16,33x. Each
The expert was asked to evaluate each test case weakness has a certain number of points associated
and detect its potential internal control weaknesses. with it, reflecting its importance. Some weaknesses
Reasons for each potential weakness were also re- have more points Ži.e., more importance. than other
quested. Then, the prototype expert system was used weaknesses.
to detect the potential weaknesses and offer reasons In arriving at an accuracy score, all points related
of such weaknesses as well. The results were then to correctly identified weaknesses are added. In order
compared. It turned out that the expert and expert to prevent the subjects from trying to detect weak-
system identified similar weaknesses for each of the nesses by guessing, they were informed at the begin-
test cases. Where there were discrepancies, the ex- ning of the experiment that one-third of all points for
pert reconsidered his responses and agreed that the inaccurately identified weaknesses will be subtracted
expert system’s responses were indeed correct. Inter- as the penalty for guessing. Non-response for a
estingly, this illustrates that an expert system can potential weakness results in neither addition nor
sometimes be useful even to an expert Že.g., to subtraction.
double check the expert’s reasoning..
5.1.2. Time per accuracy score
The time used to make correct decisions Ži.e.,
5. Examining the usefulness of the expert system time per accuracy score. was examined as a measure
of the system’s efficiency. The time per accuracy
After constructing a system with the intent to use score was computed as follows:
it to assist managers detect the potential weaknesses Time Used to Perform Task
in an internal control system, an experiment was
conducted to examine its utility. Accuracy Score
Table 3 illustrates how to calculate the Time per
5.1. Research model Accuracy Score.

The research model used to guide this research is 5.1.3. Participant satisfaction with the expert system
illustrated in Fig. 1. Two decision aid treatments A post-experiment questionnaire was used to
were used as values of the independent variable: measure the participant satisfaction with the expert

Fig. 1. Research model.

442 C. Changchit et al.r Decision Support Systems 30 (2001) 437–449

Table 3
Calculation of time per accuracy score
Examples Calculation of time per accuracy score
1 v
Mr. A correctly detects five weaknesses and incorrectly identifies five weaknesses.
v
The time used to do the case is 40 min.
v
Assuming that each weakness has three points associated with it, the accuracy score is 10 s ŽŽ5 = 3. y Ž5 = 3.r3..
v
The time per accuracy score is 4 s Ž40r10. min per accuracy score.
2 v
Mr. B detects four correct weaknesses, three incorrect weaknesses, and three non-responses.
v
The time used to do the case is 40 min.
v
Assuming that each weakness has three points associated with it, the accuracy score is 9 s ŽŽ4 = 3. y Ž3 = 3.r3..
v
The time per accuracy score is 4.4 s Ž40r9. min per accuracy score.

system. Seven-point Likert scales were used in the Hypothesis H1 was tested to examine if use of the
questionnaire. It is important to note that subjects expert system can help improve the accuracy of a
received no feedback concerning their accuracy or participant in detecting internal control weaknesses.
speed prior to completing the questionnaire. The Hypothesis H2 was tested to examine if the use of
following questions were asked to measure satisfac- the expert system can help reduce the time used by a
tion: participant in accurately detecting the internal con-
trol weaknesses. Hypotheses H3a, H3b, and H3c are
v
On a scale of 1 Žvery difficult. to 7 Žvery easy., based on the Technology Acceptance Model ŽTAM.
how difficult was it to use the expert system? w14x. The TAM suggests that a computer system’s
v
On a scale of 1 Žstrongly disagree. to 7 Žstrongly usage is determined by the behavioral intention to
agree., do you agree that using the expert sys- use such a system, which is jointly determined by the
tem helps improve the accuracy of your answers attitudes toward using the system and the perceived
to the case study? usefulness of the system. Therefore, these hypothe-
v
On a scale of 1 Žstrongly disagree. to 7 Žstrongly ses were tested to assess the attitude of the partici-
agree., do you agree that using the expert sys- pants in using the expert system.
tem helps reduce the time you used to answer
the case study? 5.2. Preparing for the experiment

Based on the foregoing research model, the exper- Aside from the expert system itself, the experi-
iment was designed to allow testing of the hypothe- ment required the preparation of several additional
ses presented in Table 4. items that were to be used by subjects in performing

Table 4
Hypotheses
Hypotheses Descriptions
H1 With the expert system, participants can detect potential weaknesses of an internal control system more accurately
than without the expert system.
H2 With the expert system, participants can accurately detect potential weaknesses of an internal control system more
quickly than without the expert system.
H3a Participants perceive that it is not difficult to use the expert system.
H3b Participants agree that using the expert system helps improve the accuracy of their answers to the case study.
H3c Participants agree that using the expert system helps reduce the time they used to answer the case study.
 C. Changchit et al.r Decision Support Systems 30 (2001) 437–449 443

the experimental task: a list of internal control weak- efficiently andror more effectively, the subjects were
nesses, three internal control case studies, and a managers in client firms of three national accounting
questionnaire. firms. Three lists of potential subjects were obtained
A list of internal control weaknesses was devel- from three international accounting firms. Then, one
oped from a review of auditing texts, accounting hundred and forty subjects were randomly selected
texts, and input from accounting professors and ex- from those lists. To gain subjects’ cooperation, the
perienced auditors. Two experienced auditors and invitation letter explained clearly the importance of
two accounting professors were asked to evaluate the study and assured confidentiality. All subjects
each weakness and assign a score to each weakness were also told that they would receive a summary of
in this list using a scale of 0 to 10 based on the this study.
degree of importance of each weakness Ž0—the least The experiment was conducted in a conference
important, 10—very important.. The average of the room and laboratory room of a college, providing an
scores for each weakness Ži.e., divided by four. was isolated and controlled environment for the study.
assigned as its importance. As described previously, There were 50 participants in the experimental group
importance scores for the weaknesses are used in Ž50% participation rate. and 15 in the control group
assessing participants’ performances in internal con- Ž37.5% participation rate.. These participants served
trol weakness recognition. as internal control decision makers. No participants
Three case studies ŽA, B, and C. were generated had either specific background in internal control
from the manipulation of several cues for detecting concepts or prior hands-on experience with an expert
the potential weaknesses in internal control systems. system. Table 5 shows the industries represented by
As with the test cases, these cues were obtained from the participants.
a review of auditing texts, accounting texts, and Participants had up to 15 years of experience with
input from accounting professors and experienced their organizations, with the average being 6.5 years.
auditors. In each case, the scenario dealt with the The percentage of their management duties ranged
adequacy of internal control over a company’s sales from 10 to 90 with an average of 48. The number of
and collection cycle. These case studies also in- employees in their organizations ranged from 3 to
cluded the background information about the com- 32,000 with an average of 814. Table 6 indicates the
pany and a partial organization chart that applies to diversity of the participants’ roles.
each scenario. Each case study contained 10 poten-
tial weaknesses in an internal control system. Three
experienced auditors and three managers were asked Table 5
Organizations’ industries
to pilot test these case studies to ensure their similar-
ity with respect to the degree of difficulty in detect- Industries No. of No. of
subjects in subjects in
ing the potential internal control weaknesses. Revi- experimental control group
sions to these case studies were made based on the group
feedback they provided. Consultant 11 1
Questionnaires were developed to gather data Legal 8
about participants’ perceptions of their experiences Manufacturing 7 1
in using the expert system, as well as their demo- Government 3 8
graphics. These questionnaires were pretested to en- Insurance 3
Banking 2
sure their clarity. Construction 2
Distribution 2
5.3. Conducting the experiment Mining 2
Transportation 2
Because the main purpose of this study is to Education 1 2
Finance 1 2
investigate whether an expert system can be devised Others 6 1
to assist managers detect potential weaknesses of Total 50 15
internal control systems in their organizations more
444 C. Changchit et al.r Decision Support Systems 30 (2001) 437–449

Table 6 the questionnaire was given to measure participant

Participants’ titles satisfaction with the use of the expert system.
Title No. of No. of In order to assess whether changes Žeither better
subjects in subjects in
or worse. in participants’ performances were due to
experimental control group
group use of the expert system or due to learning through
the practice of three consecutive cases, fifteen partic-
Finance and Accounting Manager 8 1
General Manager 6 1 ipants served as a control group. These participants
Supervisor 6 4 performed the same session cases as the experimen-
Director 4 1 tal group, but without any exposure to the expert
Senior Analyst 4 system.
Assistant Manager 3 3
A performance-based reward mechanism is cru-
Vice President 2
Tax Manager 2 cial for laboratory experiments because it creates a
Administrative Manager 2 4 more realistic environment and gives subjects incen-
Legal Manager 2 tives to perform w34x. A performance-based reward
Chief Accountant 2 system was used here to induce optimal participant
Engineer 2
behavior and reduce problems due to guessing. Par-
System Analyst 2
Lawyer 2 ticipants were told that their performances across the
Marketing Manager 1 1 series of cases would be used to calculate two scores:
Operating Manager 1 an accuracy score and a time per accuracy score.
Joint Venture Manager 1 They were informed that prizes would be given to
Total 50 15
those participants attaining the highest accuracy
scores and to those attaining the best time per accu-
racy scores.

5.4. Experimental task

6. Experimental findings
The experiment consisted of three sessions. Prior
to the experiment, the three case studies were ran- Because subjects were randomly assigned to each
domly assigned to the sessions, yielding the result of group, two ANOVA were performed on the first
Case A being assigned to Session I, Case B to session to ensure the homogeneity of subjects on
Session III, and Case C to Session II. In each internal control knowledge between the experimental
session, participants were asked to play the role of a group and the control group. The ANOVA were
manager trying to detect the potential weaknesses of performed to test for differences in Accuracy Score
the internal control system described in a case study. and Time per Accuracy Score between the groups.
The maximum time allowed for each session was At an alpha level of 0.05, no significant difference
two hours. In Session I, participants performed the was found between the groups.
case study ŽCase A. without any decision aid. Be- In order to test the hypotheses, we compared the
cause no participant has any specific background in performances of participants in Session I with their
internal control concepts, a list of possible 126 inter- performances in Session III. There were two major
nal control weaknesses was provided in this session. reasons why the performances in Session III were
The time taken to make a decision about which used instead of Session II. First, because no partici-
weakness existed was measured and the accuracy pants have a hand-on experience in using the expert
score was calculated. In Sessions II and III, partici- system, the accuracy scores in Session II might not
pants performed the second and third case studies reflect their actual competency. These scores might
ŽCase C and Case B, respectively. with the expert be reduced due to unfamiliarity with using the expert
system for each session; the time taken to make a system. Second, the time measured in Session II
decision was measured and the accuracy score was could not be considered as the actual decision time
calculated. At the end of each of Sessions II and III, because part of it might be considered as being used
Table 7
Operationalization of hypotheses
Ha Descriptions Operationalization of Hypotheses
H1 With the expert system, participants H1 0 : m I G m III ; H1 a : m I - m III .
can detect potential weaknesses of an m I s the mean of accuracy scores in Session I.
internal control system more accurately m III s the mean of accuracy scores in Session III.
than without the expert system. We predicted that, on the average, participants could better detect potential weaknesses of an internal

C. Changchit et al.r Decision Support Systems 30 (2001) 437–449

control system with the expert system than without the expert system. This prediction could be
confirmed if the null hypothesis was statistically rejected.
H2 With the expert system, participants H2 0 : m I F m III ; H2 a : m I ) m III .
can accurately detect potential m I s the mean of time per accuracy scores in Session I.
eaknesses of an internal control system m III s the mean of time per accuracy scores in Session III.
more quickly than without the expert system. We predicted that, on the average, participants can accurately detect potential weaknesses of an internal
control system more quickly with the expert system than without the expert system. This prediction
could be confirmed if the null hypothesis was statistically rejected.
H3a Participants perceive that it is not H3a 0 : m III F 4; H3a a : m III ) 4.
difficult to use the expert system. m III s the mean of responses in Session III to the question AOn a scale of 1 Žvery difficult. to 7 Žvery easy.,
how difficult was it to use the expert system?B
4 sa neutral response to the question.
We predicted that, on the average, participants would not experience difficulty in using the expert system.
This prediction could be confirmed if the null hypothesis was statistically rejected.
H3b Participants agree that using the expert H3b 0 : m III F 4; H3ba : m III ) 4.
system helps improve the accuracy of m III s the mean of responses in Session III to the question AOn a scale of 1 Žstrongly
their answers to the case study. agree., do you agree that using the expert system helps improve the accuracy of your answers to the case study?B
4 sa neutral response to the question.
We predicted that, on the average, participants would be satisfied that using the expert system helps
improve the accuracy of their answers to the case study. This prediction could be confirmed if the null
hypothesis was statistically rejected.
H3c Participants agree that using the expert H3c 0 : m III F 4; H3c a : m III ) 4.
system helps reduce the time they used to m III s the mean of responses in Session III to the question AOn a scale of 1 Žstrongly
answer the case study. agree., do you agree that using the expert system helps reduce the time you used to answer the case study?B
4 sa neutral response to the question.
We predicted that, on the average, participants would be satisfied that using the
expert system using the expert system helps reduce the time they use to answer the case study.
This prediction could be confirmed if the null hypothesis was statistically rejected.

445
446 C. Changchit et al.r Decision Support Systems 30 (2001) 437–449

to learn about interacting with the expert system. pants could detect internal control weaknesses signif-
Therefore, we felt that it was most appropriate to icantly more accurately than without the expert sys-
compare Session I vs. Session III. A t-test was tem Ž m I s 4.575 vs. m III s 27.473.. The correspond-
employed to test whether sample means were distinct ing lack of an increase in the accuracy score for the
for the hypotheses H1, H2, H3a, H3b, and H3c. control group ŽŽ m I s 8.584 vs. m III s 3.590. strongly
Table 7 illustrates the operationalization of these suggests that the increase in the mean accuracy score
hypotheses to be tested. for the experimental group was not the result of
learning that may have accrued from having the
participants perform three consecutive cases. The
7. Results and discussion t-test confirms that there was a significant difference
in participants’ accuracy in detecting an internal
The major findings for the experimental group are control weakness when they performed the task with
summarized in Table 8, which shows means and the expert system in Session III vs. without the
t-test results pertaining to the hypotheses. All hy- expert system in Session I Ž p-values 1.219ey2 2 ..
potheses are supported at a 0.01 level of signifi- Concerning hypothesis H2, Session I vs. Session
cance. Results for the control group are shown in III results show that with the expert system, partici-
Table 9 for comparison. Means for Session II are pants could accurately detect internal control weak-
also shown in both tables. Although they are not nesses significantly more quickly than without the
used in testing the hypotheses, their values are con- expert system. The mean of times divided by the
sistent with our expectation that the potential bene- mean of accuracy scores for the Session I is 25.598,
fits of the expert system would not be fully realized while the mean of times divided by the mean of
upon the subjects’ first exposure to it. accuracy scores for the session III is 3.924. The lack
Regarding hypothesis H1, Session I vs. Session of a comparable decline in mean of time per mean of
III results show that with the expert system, partici- accuracy scores in the control group Ž m I s 13.747

Table 8
Summary of findings in the experimental group
Experimental group Ž49 participants. a Session I Session II Session III Ho p-Value Session I
Participants Participants Participants vs. Session III
perform case A perform case C perform case B
with a list of with the with the
internal control expert system expert system
weaknesses
Mean of accuracy score Žthe range is y20 to 38. 4.575 15.459 27.473 H1 1.21ey2 2 ) )
Mean of time 117.102 120 107.816 nra
Mean of time per; mean of accuracy score 25.598 7.762 3.924 H2 2.182ey0 9b,) )
Mean of participants’ attitude on the 4.74 5.04 H3a 5.109ey0 7 ) )
difficulties of using the expert system
Ž1—very difficult, 7—very easy.
Mean of participants’ agreement that using 5.38 5.36 H3b 4.647ey1 0 ) )
expert system can help improve their accuracy
accuracy Ž1—strongly disagree, 7—strong agree.
Mean of participants’ agreement that using the 5.36 5.44 H3c 9.925ey1 0 ) )
expert system can help reduce their decision time
Ž1—strongly disagree, 7—strong agree.
))
Represent the significant at a s 0.01.
a
One participant had an emergency call and left while performing case II. His data was, therefore, taken out.
b
The p-value was calculated based on the adjusted time per accuracy score Žsee details in discussion of findings..
 C. Changchit et al.r Decision Support Systems 30 (2001) 437–449 447

Table 9
Summary of findings in the control group
Control group Ž15 participants. Session I Session II Session III Ho p-Value Session I
Participants Participants Participants vs. Session III
perform case A perform case C perform case B
with a list without the without the
internal control expert system expert system
weaknesses
Mean of accuracy score 8.584 5.153 .950 H1 0.052
Mean of time 118 77 67 nra
Mean of timer per; 13.747 14.942 16.962 H2 0.993
mean of accuracy score

vs. m III s 16.962. strongly suggests that the im- Recall that these perceptions are not influenced by
provement in mean of time per mean of accuracy any knowledge of actual performance in any of the
score for the experimental group was not the result experimental sessions.
of learning effects due to having the participants
perform three consecutive cases.
Because some participants obtained a 0 accuracy 8. Conclusions, limitations, and directions of fu-
score, a t-test cannot be run directly to test if there ture research
was a significant difference in the mean of time per
accuracy score between the sessions Ži.e., the time This research is just the initial investigation of the
cannot be divided by zero.. In order to solve this use of an expert system to assist managers who are
problem, a positive theoretical minimum score Ž20. not auditing experts in detecting potential internal
was added to each participant’s accuracy score. The control weaknesses. The primary finding of this study
result is referred to as an adjusted time per accuracy is that it is feasible to build such an expert system.
score. The result of conducting a t-test using these The managers participated in this study have demon-
adjusted time per accuracy scores confirms that there strated a high interest in using the system. Installing
was a significant difference in the time that partici- this kind of system might let organizations save time
pants used to accurately detect internal control weak- and money by allowing weaknesses in internal con-
nesses with expert system support vs. without the trol systems to be detected and solved more quickly.
expert system Ž p-values 2.182ey0 9 .. This type of system could help in maintaining an
Regarding hypotheses H3a, H3b, and H3c, the effective internal control system, thus providing more
participants’ answers to the post-experiment ques- reliable accounting data and more safeguarding of
tions reveal satisfaction with using the expert system. assets. Such a system can also be beneficial for
The results show their attitudes toward the using of auditing firms by facilitating more reliable internal
the expert system as follows: control in client firms, thereby reducing planned
detection risk Ži.e., less work and time..
v
It is not difficult to use the expert system Ž m III The scope of this research study may limit the
s 4.89, p-values 5.109ey0 9 .. generalizability of the results in several respects.
v
They agree that using the expert system helps First, this research concentrates only on the evalua-
improve the accuracy of their answer to the case tion of controls commonly found in sales and collec-
study Ž m III s 5.37, p-values 4.647ey1 0 .. tion cycle. Second, it investigates internal control
v
They agree that using the expert system helps systems commonly found in the merchandising in-
reduce the time they used to answer the case dustry. It is not expected to handle novel Žuncom-
study Ž m III s 5.40, p-values 9.925ey1 0 .. monly different. accounting systems. Third, the
448 C. Changchit et al.r Decision Support Systems 30 (2001) 437–449

knowledge of the expert system developed for this nique and a Case Study Approach, PhD dissertation, Virginia
study is based primarily on one auditor who is a Polytechnic Institute and State University, 1991.
w8x R.H. Bonczek, C.W. Holsapple, A.B. Whinston, Foundations
partner of an international accounting firm. The re- of Decision Support Systems, Academic Press, New York,
sulting system closely represents his reasoning about 1981.
internal control evaluation. Thus, the system’s w9x J.E. Boritz, The effect of information presentation structures
knowledge may be firm-specific or expert-specific. on audit planning and review judgments, Contemp. Account.
These limitations point to directions in which the Res. 8 Ž1. Ž1985. 193–218.
w10x C.E. Brown, M.E. Phillips, Expert systems for internal audit-
research presented here can be extended by future ing, Intern. Auditor 48 Ž8. Ž1991. 23–28.
investigations. w11x Committee of Sponsoring Organizations of Treadway Com-
The future research might try to investigate the mission, Internal Control Integrated Framework, American
results of using the expert systems over a long period Institute of CPA, New York, 1992.
w12x B.K. Cummings, N.G. Apostolou, Expert systems in audit-
of time Že.g., conducting additional sessions 1 week
ing: an emerging technology, Intern. Auditing 3 Ž2. Ž1987.
or month after the first three sessions.. Another 3–10.
research avenue might investigate the use of such an w13x W. Cummings, J. Lauer, R. Baker, Expert systems in internal
expert system as a training tool, instead of as the auditing, Intern. Auditing 4 Ž1. Ž1988. 49–64.
direct decision aids presented in this paper. Re- w14x F.D. Davis, R.D. Bagozzi, P.R. Warshaw, User acceptance of
searchers might try to incorporate additional transac- computer technology: a comparison of two theoretical mod-
els, Manage. Sci. 35 Ž8. Ž1989. 982–1003.
tion cycles, industries, or other auditing functions w15x J. Durkin, Expert systems: a view of the field, IEEE Expert
into the expert system. They might attempt to de- 11 Ž2. Ž1996. 56–63.
velop additional expert systems by acquiring the w16x M.M. Eining, P.B. Dorr, The impact of expert system usage
expertise from an internal auditor instead of the on experiential learning in an auditing setting, J. Inf. Syst. 5
Ž1. Ž1991. 1–16.
external auditor. They might try to acquire the exper-
w17x C.L. Foltin, L. Garceau, Beyond expert systems: neural
tise from multiple auditors and then conduct experi- networks in accounting, Natl. Public Account. 41 Ž1. Ž1996.
ments similar to the one reported here. Finally, re- 26–30.
searchers might investigate the feasibility of integrat- w18x C.L. Foltin, M.L. Smith, Accounting expert systems, CPA J.
ing this kind of expert system with the company’s 64 Ž11. Ž1994. 46–53.
w19x V.M. Gadh, R. Krishnan, J.M. Peters, Modeling internal
databases.
control and their evaluation, Auditing: J. Pract. Theory 12
Ž1993. 113–129, Supplement.
w20x G.F. Gal, Using Auditor Knowledge to Formulate Data
Model Constraints: An Expert System for Internal Control
Evaluation, PhD dissertation, Michigan State University,
References 1985.
w21x L.E. Graham, J. Damens, G.V. Ness, Developing risk ad-
visor SM : an expert system for risk identification, Auditing: J.
w1x American Institute of Certified Public Accountants. Codifica- Pract. Theory 10 Ž1. Ž1991. 69–96.
tion of Statements on Auditing Standards, American Institute w22x J.V. Hansen, W.F. Messier Jr., A preliminary investigation of
of Certified Public Accountants, New York, 1994. EDP-XPERT, Auditing: J. Pract. Theory 6 Ž1. Ž1986. 109–
w2x American Institute of Certified Public Accountants, Codifica- 123.
tion of Statements on Auditing Standards, American Institute w23x C.W. Holsapple, V.S. Raj, An exploratory study of two KA
of Certified Public Accountants, New York, 1996. methods, Expert Syst. 11 Ž2. Ž1994. 77–88.
w3x A.A. Arens, J.K. Loebbecke, Auditing: An Integrated Ap- w24x C.W. Holsapple, A.B. Whinston, Business Expert Systems,
proach, 6th edn., Prentice-Hall, Englewood Cliffs, NJ, 1994. Irwin, Homewood, IL, 1987.
w4x B. Back, Validating an expert system for financial statement w25x International Standard on Auditing, Risk Assessment and
planning, J. Manage. Inf. Sys. 10 Ž3. Ž1994. 157–177. Internal Control, International Federation of Accountants,
w5x A.D. Bailey Jr., G.L. Duke, J. Gerlach, C. Ko, R. Meservy, New York, NY, 1991.
A.B. Whinston, TICOM and the analysis of internal controls, w26x E.G. Jancura, Expert systems: an important new technology
Account. Rev. 60 Ž2. Ž1985. 186–211. for accountants, Woman CPA 52 Ž2. Ž1990. 25–28.
w6x A.D. Baily, R.P. McAfee, A.B. Whinston, An application of w27x S.R. McDuffie, D. Oden, E.P. Porter, Tax expert systems and
complexity theory to the analysis of internal control systems, future development, CPA J. 64 Ž1. Ž1994. 73–75.
Auditing: J. Pract. Theory 1 Ž1. Ž1981. 38–52. w28x R.D. Meservy, A.D. Bailey Jr., P.E. Johnson, Internal Con-
w7x A.A. Baldwin-Morgan, The Impact of Expert Systems on trol Evaluation: a computational model of the review process,
Auditing Firms: An Investigation Using the Delphi Tech- Auditing: J. Pract. Theory 6 Ž3. Ž1986. 44–74.
 C. Changchit et al.r Decision Support Systems 30 (2001) 437–449 449

w29x W.F. Messier, J.V. Hansen, Artificial Intelligence in Ac- w33x V.L. Smith, Microeconomic systems as experimental science,
counting and Auditing, Marcus Wiener, New York, 1989. Am. Econ. Rev. 72 Ž5. Ž1982. 923–955.
w30x D.E. O’Leary, Methods of validating expert systems, Inter- w34x A.S. Vinze, V. Karan, U.S. Murthy, A generalizable knowl-
faces 18 Ž6. Ž1988. 72–79. edge-based framework for audit planning expert systems, J.
w31x D.E. O’Leary, P.R. Watkins, Review of expert systems in Inf. Syst. Ž1991. 78–91 ŽFall 1991..
auditing, Expert Syst. Rev. Business Account. 9 Ž2. Ž1989. w35x R. Weber, Information Systems Control and Audit, Prentice-
3–22. Hall, Upper Saddle River, NJ, 1999.
w32x R. Sharda, S.H. Barr, J.C. McDonnell, Decision support
system effectiveness: a review and an empirical test, Man-
age. Sci. 34 Ž2. Ž1988. 139.