Journal of the American Medical Informatics Association, 24(e1), 2017, e55–e60

doi: 10.1093/jamia/ocw100
Advance Access Publication Date: 7 August 2016
Research and Applications

Research and Applications

Comparison of heuristic and cognitive walkthrough

usability evaluation methods for evaluating health
information systems
Reza Khajouei,1,2 Misagh Zahiri Esfahani,3 and Yunes Jahani4
1
Medical Informatics Research Center, Institute for Futures Studies in Health, Kerman University of Medical Sciences, Kerman,
Iran, 2Department of Health Information Management and Technology, Faculty of Management and Medical Information Scien-
ces, Kerman University of Medical Sciences, Kerman, Iran, 3Regional Knowledge Hub and World Health Organization Collaborat-
ing Centre for HIV Surveillance, Institute for Futures Studies in Health, Kerman University of Medical Sciences, Kerman, Iran, and
4
Department of Biostatistics and Epidemiology, School of Public Health, Kerman University of Medical Sciences, Kerman, Iran

Corresponding Author: Misagh Zahiri Esfahani, Regional Knowledge Hub and WHO Collaborating Centre for HIV Surveil-
lance, Institute for Futures Studies in Health, Kerman University of Medical Sciences, Haft Bagh-e-Alavi Highway, PO
Box 7616911320, Kerman, Iran 76175. E-mail: [email protected]; Fax: þ98-(0)-34-31325409
Received 16 October 2015; Revised 22 May 2016; Accepted 24 May 2016

ABSTRACT
Objective: There are several user-based and expert-based usability evaluation methods that may perform differ-
ently according to the context in which they are used. The objective of this study was to compare 2 expert-
based methods, heuristic evaluation (HE) and cognitive walkthrough (CW), for evaluating usability of health
care information systems.
Materials and methods: Five evaluators independently evaluated a medical office management system using
HE and CW. We compared the 2 methods in terms of the number of identified usability problems, their severity,
and the coverage of each method.
Results: In total, 156 problems were identified using the 2 methods. HE identified a significantly higher number
of problems related to the “satisfaction” attribute (P ¼ .002). The number of problems identified using CW con-
cerning the “learnability” attribute was significantly higher than those identified using HE (P ¼ .005). There was
no significant difference between the number of problems identified by HE, based on different usability attrib-
utes (P ¼ .232). Results of CW showed a significant difference between the number of problems related to
usability attributes (P < .0001). The average severity of problems identified using CW was significantly higher
than that of HE (P < .0001).
Conclusion: This study showed that HE and CW do not differ significantly in terms of the number of usability
problems identified, but they differ based on the severity of problems and the coverage of some usability attrib-
utes. The results suggest that CW would be the preferred method for evaluating systems intended for novice
users and HE for users who have experience with similar systems. However, more studies are needed to sup-
port this finding.

Key words: health information systems, heuristic evaluation, cognitive walkthrough, user-computer interface, comparison

C The Author 2016. Published by Oxford University Press on behalf of the American Medical Informatics Association.
V
All rights reserved. For Permissions, please email: [email protected]
e55
e56 Journal of the American Medical Informatics Association, 2017, Vol. 24, No. e1

INTRODUCTION focuses on the learnability of a system for new users. In this method,
evaluators step through a user interface by exploring every step
Many health care information systems have been developed to pro-
needed to complete a task. Meanwhile, they examine how easy it is
mote the quality of health care. Computer-based patient records is
for new users to accomplish tasks with the system. CW can be
one system that can manage patient information, save clinician time,
applied to study complex user interactions and goals.9,17
and improve safety in the health care domain.1 Because of its posi-
tive effect on patient care and the quality of outcomes, use and
adoption of this system have increased over time.2–4 However, there System
are still some barriers to the successful interaction of users with this To compare the HE and CW methods, we evaluated a medical office
system and subsequently its complete adoption. management system, called Clinic 24, as a sample of computer-
Researchers5,6 have shown that usability problems are among based patient records. This system was developed by Rayan Pardaz
the significant barriers to adoption of health information technol- Corporation and is used in 71 medical offices throughout Iran. To
ogy, and they can negatively affect practitioners’ decision making, facilitate the evaluation process, the system was installed on the
time management, and productivity.6 This can lead to user fatigue researchers’ computer with the permission of the software provider.
and confusion and subsequently to withdrawal or rejection of the The system has 2 parts, secretarial (used by physicians’ secretaries)
systems. Hence, evaluation of health information systems seems to and clinical (used by physicians). Given the importance and higher
be essential in order to make user interaction more effective. Several relevance of the clinical part to the health care domain, the focus of
methods exist for evaluating computer-based information systems. this study was on this part. Physicians document the treatment proc-
They can be categorized into 2 main groups: user-based and expert- ess in the system by either using a light pen or selecting from prede-
based methods. Heuristic evaluation (HE) and cognitive walk- fined options in the system. In this part, physicians can document
through (CW) are 2 well-known expert-based methods that can the treatment process based on the Subjective, Objective, Assess-
identify a large number of problems using small amounts of finan- ment, and Plan (SOAP) documentation method. This part provides
cial and time resources.7,8 HE is guided by heuristic principles5 to functionality to document the chief complaint, diagnosis, and drug
identify user interface designs that violate these principles. CW eval- prescription(s) for each patient.
uates the degree of difficulty to accomplish tasks using a system to
determine the actions and goals needed to accomplish each task.9 Participants
The question now arises: which of these methods performs better in Since 3 to 5 evaluators are considered sufficient to perform HE and
terms of identifying user interaction problems with health informa- CW evaluations,5,18 5 evaluators were recruited in this study to eval-
tion systems, and what are the differences? uate Clinic 24. The evaluators had a background in health informa-
Based on a search by the authors, only 1 study10 used HE and tion technology and received theoretical and practical training about
CW methods synchronously in the domain of health care. The goal the HE and CW evaluation methods. All evaluators had a minimum
of this study was to develop a hybrid method, and it did not com- of 3 years’ evaluation experience, and 3 of them had already been
pare the performance of these 2 expert-based methods for identify- engaged in other domestic usability evaluation projects. They agreed
ing usability problems. Other comparative studies on HE and/or to conduct the evaluation and completed informed consent forms in
CW compared 1 of these 2 usability methods with usability testing advance. The study was approved by the ethics committee of Ker-
methods. For example, among the studies in the field of health care, man University of Medical Sciences (Ref. No. K/93/693).
Yen and Bakken11 and Thyvalikakath et al.12 compared the HE
method with user-testing methods. In another study, Khajouei
et al.13 compared CW with the think-aloud method (a type of user- Data collection
testing method). In HE, each evaluator examined the conformity of the user interface
Since the performance of these 2 expert-based methods (HE and of Clinic 24 to Nielsen 10 heuristic principles (Table 1). Any mis-
CW) has not been compared in the domain of health care, the objec- match was identified as a usability problem.
tive of this study was to compare these 2 methods to identify usabil- In CW the user interface of the system was evaluated based on
ity problems of health information systems. Also, given that each the methodology proposed by Polson and Lewis.9 This evaluation
usability problem affects users in a different way,14 we aimed to was carried out using 10 scenarios. Each scenario consisted of a set
compare the 2 methods in terms of the different types of identified of tasks that evaluators had to perform using the system. The scenar-
usability problems. ios were carefully designed, in consultation with several physicians
and the designers of the system, to be as representative of the physi-
cians’ daily work as possible. Figure 1 shows 3 examples of the sce-
narios and their corresponding tasks.
MATERIALS AND METHODS
To perform the CW evaluation, we first determined initial user
Evaluation methods goals and subgoals based on each scenario and made an action-
In this study, we compared the heuristic evaluation (HE) and cogni- sequence list as described by Polson et al.9 Independent evaluators
tive walkthrough (CW) methods for evaluating the usability of then systematically stepped through the system by examining each
health information systems. Both are inspection (expert-based) task, noting: (1) user goals, (2) user subgoals, (3) user actions, (4)
methods conducted without the involvement of users.15 However, system responses, and (5) potential user interaction problems. Each
the main focus and process of applying these 2 methods are fairly evaluator independently provided a list of usability problems with
different. HE is done by expert evaluators examining the design of a their descriptions and corresponding screenshots in a Word file. Ver-
user interface and judging its compliance with a list of predefined bal comments of evaluators were transcribed by a coordinator. Eval-
principles (heuristics). These principles are used as a template, help- uators were also encouraged to write down their extra comments.
ing the evaluators to identify the potential problems users may Certain parts of the system that are rarely used by physicians
encounter.5,16 CW is a task-oriented and structured method that were not evaluated. To prevent bias, HE was done on all parts that
Journal of the American Medical Informatics Association, 2017, Vol. 24, No. e1 e57

Table 1. Nielsen 10 heuristic principles5,19

Heuristic principle Definition

1 Visibility of system status Users should be informed about what is going on with the system through appropriate feedback.
2 Match between system and the The image of the system perceived by users and presentation of information on screen should match
real world the model users have about the system.
3 User control and freedom Users should not have the impression that they are controlled by the system.
4 Consistency and standards Users should not have to wonder whether different words, situations, or actions mean the same thing.
Design standards and conventions should be followed.
5 Error prevention It is always better to design interfaces that prevent errors from happening in the first place.
6 Recognition rather than recall The user should not have to remember information from one part of the system to another.
7 Flexibility and efficiency of use Both inexperienced and experienced users should be able to customize the system, tailor frequent
actions, and use shortcuts to accelerate their interaction.
8 Aesthetic and minimalist design Any extraneous information is a distraction and a slowdown.
9 Help users recognize, diagnose, and Error messages should be expressed in plain language (no codes), precisely indicate the problem,
recover from errors and constructively suggest a solution.
10 Help and documentation System should provide help when needed.

Attribute Definition ISO* / Nielsen

Effectiveness How well do the users achieve the goals they set out ISO 9241
to accomplish using the system?
Efficiency What amount of resources (e.g. time and mental effort) ISO 9241- Nielsen
do users need in order to achieve their goals?

Satisfaction How pleasant is it to use the system? ISO 9241- Nielsen

Learnability How easy is it for users to accomplish basic tasks the Nielsen
first time they encounter the system?

Memorability When users return to the system after a period of not Nielsen
using it, how easily can they reestablish proficiency?

Errors How many errors do users make, how severe are Nielsen
these errors, and how easily can they recover from the
errors?

*International Standard Organization

Figure 2. Usability attributes extracted from the ISO and Nielsen

definitions14,22.

Figure 1. Examples of scenarios and their corresponding tasks. The master lists were distributed among evaluators, and they
independently determined the severity of the problems. The problem
CW was performed on (those covered by the 10 scenarios). The eval- severity was assigned based on a combination of 3 factors: frequency
uation was done in 2 rounds. To increase the validity of the results of problem, potential impact of problem on user, and persistence of
and prevent any learning effect, we counterbalanced the order of the problem every time user faces the same situation. Using these fac-
evaluators for each method. The first method for each evaluator was tors, evaluators rated the severity of each problem according to the
specified randomly. In the first round of the evaluation, 3 evaluators Nielsen 5-scale rating as follows: 0 ¼ not a problem at all,
performed HE and 2 evaluators CW. After a washout period of 4 1 ¼ cosmetic problem only, 2 ¼ minor usability problem, 3 ¼ major
weeks, the order of the evaluators was reversed. usability problem, 4 ¼ usability catastrophe.20,21 The absolute
severity of each problem was determined by averaging the severity
scores assigned by different evaluators to that problem.
Analysis In this study, we compared the 2 methods in terms of the number
The collected data from independent evaluations were reviewed and of identified usability problems, the severity of problems, and the
compared in joint meetings by the 5 evaluators. We held 3 formal coverage of each method. To compare the coverage of the 2 usability
joint meetings, each lasting approximately 3 hours with short evaluation methods, problems were categorized based on a combi-
breaks, to analyze both the HE and CW results and to categorize the nation of usability attributes proposed by the International Standard
identified problems. Another meeting was scheduled to summarize Organization (ISO)14 and Nielsen22 (Figure 2). In the ISO definition,
and calculate the average severity of the problems (45 minutes). effectiveness, efficiency, and satisfaction are identified as key attrib-
Each meeting was coordinated by one of the authors. In the meet- utes of usability.14 Nielsen also put forward 5 usability attributes:
ings, by merging identical problems identified by different evalua- learnability, efficiency, memorability, errors, and satisfaction.22 In
tors, a master list of unique problems was provided for each line with the usability attribute definitions in Figure 2, the decision
method. Every identified problem was discussed and any disagree- to assign problems to each category was made based on analyses of
ment was resolved by consensus. Whenever no agreement was the evaluator’s verbal feedback noted by the coordinator or them-
reached, we regarded an issue as a usability problem if confirmed by selves, the written descriptions of problems, and the screenshots.
at least 3 reviewers. Examples of problems, as categorized based on the violated
e58 Journal of the American Medical Informatics Association, 2017, Vol. 24, No. e1

Figure 3. Examples of problems, categorized based on the violated attributes.

Figure 4. The number of usability problems in both methods in terms of

Table 2. Number of problems and evaluators per method in the 2 usability attributes.
rounds of evaluation

Round of Heuristic evaluation Cognitive walkthrough

evaluation Table 3. Comparison of the 2 methods based on categories of
No. of No. of No. of No. of problems
evaluators problems evaluators problems
Usability problems HE* CW**
Round 1 3 47 2 37 N (%) N (%)
Round 2 2 45 3 27 83 (53) 73 (47)

Usability attributes
Effectiveness 9 (47) 10 (53)
attributes, are shown in Figure 3. Although a specific problem may Error 10 (77) 3 (23)
affect more than 1 usability attribute, we assigned each problem to Satisfaction 21 (81) 5 (19)
Learnability 13 (28) 33 (72)
the most appropriate attribute category by consensus.
Memorability 14 (50) 14 (50)
Data were analyzed using SPSS (version 20, SPSS Inc., Chicago,
Efficiency 16 (67) 8 (33)
IL, USA). A Chi-square test was used to compare the number of
usability problems identified by each method and the coverage of *Heuristic Evaluation.
each method. Also we used Chi-square to check the potential carry- **Cognitive Walkthrough.
over effects by comparing the number of problems identified in the 2
rounds of the evaluation per method. The mean severity of problems
identified by the 2 methods was compared using a Mann-Whitney
test. In this study, we used a significance level of 0.05.
Figure 4. There was no significant difference between the number
of problems identified using HE, based on the different usability
attributes (P ¼ .232). Results of CW showed a significant differ-
RESULTS ence between the number of problems related to different usabil-
The HE and CW methods identified 92 and 64 usability problems, ity attributes (P < .0001). The CW method performed notably
respectively. Table 2 shows the number of problems identified by better in identifying “learnability” problems (45% of CW
each method and the number of evaluators per method in the 2 results).
rounds of evaluation. There were no significant differences between Table 3 compares the 2 methods in terms of the total number
the numbers of problems identified in the 2 rounds of evaluation for of identified problems and the problems related to each usability
each method (P > .05), indicating that the carryover learning effects attribute. There was no significant difference between the 2 meth-
were small or nonexistent. ods based on total number of problems and problems related to
Twenty-five out of these 156 problems were identified identically the “effectiveness,” “error,” “memorability,” and “efficiency”
by both methods. Removing duplications between the results of the attributes (P > .05). HE identified a significantly higher number
2 evaluations left 131 unique problems. Each of the methods could of problems related to the “satisfaction” attribute than CW did
identify <70% of the total unique problems. HE detected 83 prob- (P ¼ .002). The number of problems identified by CW concerning
lems encompassing 63% (95% confidence interval [CI], 0/55-0/7) the “learnability” attribute was significantly higher than those
and CW identified 73 problems encompassing 56% (95% CI, 0/47- identified by HE (P ¼ .005).
0/64) of the total unique problems. Table 4 compares the severity of the problems identified by the 2
The number and percentage of problems identified using methods. The average severity of the problems identified by CW
each of the methods in terms of usability attributes is shown in was significantly greater than that of HE.
Journal of the American Medical Informatics Association, 2017, Vol. 24, No. e1 e59

Table 4. Comparing severity of the problems identified by the 2 These problems can be identified by CW only if they potentially
evaluation methods impair the process of doing a task. However, more studies on a
wider range of applications with more participants are required to
Mann-Whitney test Median (IQR) Mean 6 SD Method
determine the many factors affecting the results of a usability evalu-
Z ¼ 3.82 P < .0001 2.2 (1) 0.58 6 1.93 HE ation study.
2.4 (0.8) 0.54 6 2.32 CW Previous studies8,28 used the number and severity of identified
usability problems to compare usability evaluation methods. In this
study, besides these 2 criteria, we also used a combination of usabil-
DISCUSSION ity attributes proposed by ISO and Nielsen14,22 to compare the cov-
Principal findings erage of usability attributes per method.
The results of this study showed no significant difference between HE
and CW in terms of the total number of identified usability problems. Meaning of the study and directions for future research
However, these methods differ from each other in detecting problems Our study showed that CW found significantly more usability prob-
concerning 2 usability attributes, learnability and satisfaction. They lems concerning the “learnability” attribute, while HE found more
also differ based on the severity of identified problems. concerning the “satisfaction” attribute. It has also been emphasized
To our knowledge, this is the first study comparing HE and CW in the relevant literature29 that, compared to other evaluation meth-
for evaluating health information systems. Hence, we can discuss ods, CW has a stronger focus on learnability issues. Learnability is
our results only in relation to studies evaluating information systems the attribute that mainly affects novice users when trying to use a
in other domains. Although in our study the HE method found more system for the first time.34 Satisfaction is the attribute that mainly
usability problems than CW, this difference was not statistically sig- can be judged by users with experience with a system.35 These find-
nificant. Jeffries et al.,8 in a study comparing 4 usability evaluation ings suggest that CW would be the preferred method for evaluating
methods, including HE and CW, showed that the HE method identi- systems intended for novice users and HE for users with experience
fied a significantly larger number of problems. In a study by Double- with similar systems. However, more studies of this type on different
day et al.,23 comparing HE with a user-testing method, most of the systems are required to confirm this finding and to determine the
usability problems were found by the HE method. However, the coverage of other attributes by each method.
results of HE depended to a large extent on the competency and In this study, the mean severity of the problems identified using
expertise of evaluators.5,24–27 In this study, the same group of eval- CW was significantly greater than that of problems identified using
uators carried out both HE and CW, to prevent any bias on the part HE. Consistent with our results, Jeffries et al.8 showed that CW
of the evaluators. In the previous studies,19,28 2 different groups of detects more severe problems than HE. The greater severity of prob-
evaluators ran the studies, and the evaluators of HE were user inter- lems in the CW evaluation can be explained as follows. HE checks
face experts. whether the design of a user interface conforms to a limited number
Some studies18,29 concluded that since HE is guided by a set of of predefined principles without taking into account the tasks to be
principles, evaluators can freely step through the user interface to performed by potential users. These principles are general and may
find violations without paying attention to the steps required to find some superficial and common problems that may not bother all
complete a task; hence, it can catch the problems that CW may miss. users, while CW is a task-oriented method with explicit assumptions
CW is led by a series of scenarios that should cover all possible about users and their tasks. CW, therefore, can detect problems that
actions a representative user takes to complete a task.9,30 It is there- may hinder the accomplishment of a task or affect a specific type of
fore recommended to use an adequate number of scenarios to ensure user. This makes CW a better choice for evaluating mission-critical
all users’ tasks are covered. While it is not clear how comprehensive systems such as health information systems.
the scenarios of a CW study are, some of the previous studies may The results of this study shed light on the potential differences
not have used a sufficient number of scenarios to cover all the users’ between the HE and CW methods and can help evaluators of health
tasks.31,32 information systems to select the appropriate methods based on
their users and systems.

Strengths and weaknesses

In this study, we developed 10 scenarios in consultation with a
group of physicians (end users) and developers of the system to
CONCLUSION
ensure that they were representative of all users’ interactions with This study showed that HE and CW do not differ significantly in
the system in a real setting. This may have minimized the difference terms of the number of usability problems identified by each
between the number of problems identified by HE and CW com- method. However, CW works significantly better for identifying
pared to the results of previous studies. Moreover, based on our usability problems that affect learnability of the system, and HE per-
results and the results of previous studies,18,29 the design of a system forms better for detecting problems that result in user dissatisfac-
can also affect the results of an evaluation study using a specific tion. Based on this, CW may be a better choice for evaluating
method. There are many approaches to designing information sys- systems designed for novice users and HE for users who have pre-
tems. One designer may follow a series of predefined principles and vious experience with a similar system. However, more studies of
standards addressing issues such as layout, appearance, and consis- this type on a wide range of systems can support our results. More-
tency of interface design, while another may focus on the tasks to be over, the potential of CW to identify problems of higher severity
accomplished by the system (user actions and system responses).33 makes it suitable for evaluating mission-critical systems. The results
This may somewhat affect the results of some of the compared eval- of expert-based methods such as HE and CW are somewhat depend-
uation methods. For example, in this study, HE identified many ent on the evaluators’ expertise and the number and comprehensive-
problems negatively affecting the aesthetic design of the system. ness of materials such as study scenarios.
e60 Journal of the American Medical Informatics Association, 2017, Vol. 24, No. e1

AUTHOR CONTRIBUTIONS 14. Abran A, Khelifi A, Suryn W, Seffah A. Consolidating the ISO usability
models. Proceedings of 11th International Software Quality Management
R.K. and M.Z. contributed to the conception and design of the Conference. Citeseer. 2003;2003:23–25.
study, acquisition and interpretation of the data, and drafting the 15. Mack RL, Nielsen J. Usability Inspection Methods. New York: Wiley &
paper. Y.J. was primarily responsible for the statistical analysis of Sons; 1994.
the data. All 3 authors read and approved the final version of the 16. Nielsen J, Molich R. Heuristic evaluation of user interfaces. Proceedings
article submitted. of the SIGCHI Conference on Human Factors in Computing Systems.
ACM. 1990;1990:249–256.
17. Wharton C, Rieman J, Lewis C, Polson P. The cognitive walkthrough
CONFLICT OF INTEREST method: a practitioner’s guide. Usability Inspection Methods. Wiley &
Sons. 1994;1994:105–140.
The authors declare that there is no conflict of interest in this study.
18. Nielsen J. Usability inspection methods. Conference Companion
on Human Factors in Computing Systems. ACM.
1995;1995:377–378.
ACKNOWLEDGMENTS 19. Khajouei R, Peek N, Wierenga P, Kersten M, Jaspers MW. Effect of prede-
The authors would like to thank Rayan Pardaz Corporation for installation fined order sets and usability problems on efficiency of computerized med-
of the software on the researchers’ computer. We are grateful to Elnaz Mova- ication ordering. Int J Med Inform. 2010;79(10):690–698.
hedi, Sadriyeh Hajesmail Gohari, Zahra Karbasi, and Elham Saljooghi for 20. Nielsen J. Severity ratings for usability problems. Nielsen Norman Group.
their contributions to the evaluation. 1995. Available at: https://www.nngroup.com/articles/how-to-rate-the-
severity-of-usability-problems/. Accessed June 10, 2016.
21. Nielsen J. Usability Engineering. Boston, MA: Academic Press.
REFERENCES 1993;102–105.
1. Erstad TL. Analyzing computer-based patient records: a review of litera- 22. Buchanan S, Salako A. Evaluating the usability and usefulness of a digital
ture. J Healthc Inform Manag. 2003;17(4):51–57. library. Library Rev. 2009;58(9):638–651.
2. Lehmann CU, O’Connor KG, Shorte VA, Johnson TD. Use of electronic 23. Doubleday A, Ryan M, Springett M, Sutcliffe A. A comparison of usabil-
health record systems by office-based pediatricians. Pediatrics. ity techniques for evaluating design. Proceedings of the 2nd Conference
2015;135(1):e7–e15. on Designing Interactive Systems: Processes, Practices, Methods, and
3. Xierali IM, Hsiao CJ, Puffer JC, et al. The rise of electronic health record Techniques. ACM. 1997;1997:101–110.
adoption among family physicians. Ann Family Med. 2013;11(1):14–19. 24. Choi J, Bakken S. Web-based education for low-literate parents in Neona-
4. Hsiao C-J, Hing E. Use and Characteristics of Electronic Health Record tal Intensive Care Unit: development of a website and heuristic evaluation
Systems Among Office-Based Physician Practices, United States, 2001- and usability testing. Int J Med Inform. 2010;79(8):565–575.
2012. Huntsville, MD: US Department of Health and Human Services, 25. Hripcsak G, Wilcox A. Reference standards, judges, and comparison sub-
Centers for Disease Control and Prevention, National Center for Health jects. J Am Med Inform Assoc. 2002;9(1):1–15.
Statistics (NCHS) Data Brief, No 111; 2012. 26. Khajouei R, Ahmadian L, Jaspers MW. Methodological concerns in us-
5. Nielsen J. Finding usability problems through heuristic evaluation. Pro- ability evaluation of software prototypes. J Biomed Inform.
ceedings of the SIGCHI Conference on Human Factors in Computing Sys- 2011;44(4):700–701.
tems. ACM. 1992;1992:373–380. 27. Hertzum M, Jacobsen NE. The evaluator effect: A chilling fact about us-
6. Simon SR, Kaushal R, Cleary PD, et al. Correlates of electronic health re- ability evaluation methods. Int J Hum Comput Interact.
cord adoption in office practices: a statewide survey. J Am Med Inform 2003;15(1):183–204.
Assoc. 2007;14(1):110–117. 28. Karat C-M, Campbell R, Fiegel T. Comparison of empirical testing and
7. Polson P, Rieman J, Wharton C, Olson J, Kitajima M. Usability inspection walkthrough methods in user interface evaluation. Proceedings of the
methods: rationale and examples. The 8th Human Interface Symposium SIGCHI Conference on Human Factors in Computing Systems. ACM.
(HIS92). Kawasaki, Japan. 1992;1992:377–384. 1992;1992:397–404.
8. Jeffries R, Miller JR, Wharton C, Uyeda K. User interface evaluation in 29. Jaspers MW. A comparison of usability methods for testing interactive
the real world: a comparison of four techniques. Proceedings of the SIG- health technologies: methodological aspects and empirical evidence. Int J
CHI Conference on Human Factors in Computing Systems. ACM. Med Inform. 2009;78(5):340–353.
1991;1991:119–124. 30. Blackmon M. Cognitive walkthrough. Encyclopedia Hum Comput Inter-
9. Polson PG, Lewis C, Rieman J, Wharton C. Cognitive walkthroughs: a act. 2004;2:104–107.
method for theory-based evaluation of user interfaces. Int J Man-machine 31. Karahoca A, Bayraktar E, Tatoglu E, Karahoca D. Information system de-
Stud. 1992;36(5):741–773. sign for a hospital emergency department: a usability analysis of software
10. Kushniruk AW, Monkman H, Tuden D, Bellwood P, Borycki EM. Inte- prototypes. J Biomed Inform. 2010;43(2):224–232.
grating heuristic evaluation with cognitive walkthrough: development of a 32. Khajouei R, de Jongh D, Jaspers MW. Usability evaluation of a computer-
hybrid usability inspection method. Stud Health TechnolI Inform. ized physician order entry for medication ordering. MIE.
2014;208:221–225. 2009;2009:532–536.
11. Yen PY, Bakken S. A comparison of usability evaluation methods: heuristic 33. Huenerfauth MP. Design approaches for developing user-interfaces acces-
evaluation versus end-user think-aloud protocol: an example from a web- sible to illiterate users. Proceedings of the 18th National Conference on
based communication tool for nurse scheduling. AMIA Annual Symposium Artificial Intelligence. In Intelligent and Situation-Aware Media and Pre-
Proceedings/AMIA Symposium AMIA Symposium. 2009;2009:714–718. sentations Workshop (AAAI-02), Edmonton, Alberta, Canada. 2002.
12. Thyvalikakath TP, Monaco V, Thambuganipalle H, Schleyer T. Compa- Available at: http://www.aaai.org/Papers/Workshops/2002/WS-02-08/
rative study of heuristic evaluation and usability testing methods. Stud WS02-08-005.pdf. Accessed June 9, 2016.
Health Technol Inform. 2009;143:322. 34. Faulkner L, Wick D. Cross-user analysis: benefits of skill level comparison
13. Khajouei R, Hasman A, Jaspers MW. Determination of the effectiveness in usability testing. Interact Comput. 2005;17(6):773–786.
of two methods for usability evaluation using a CPOE medication order- 35. Lindgaard G, Dudek C. What is this evasive beast we call user satisfac-
ing system. Int J Med Inform. 2011;80(5):341–350. tion? Interact Comput. 2003;15(3):429–452.