Usability Testing for Educational Computer Game Using Observation Method

Norizan Mat Diah 1, Marina Ismail2, Suzana Ahmad3 and Mohd Khairulnizam Md Dahari4
Faculty of Computer and Mathematical Sciences
Universiti Teknologi MARA Shah 40450 Alam Selangor, Malaysia
[email protected],[email protected],[email protected],[email protected]

Abstract— For most children the first contact with the usability lifecycle in their software development lifecycle.
computer is through some sort of educational game. Therefore, it Various usability models and methods have been proposed to
is important that computer games for children are well-designed help in comparing the usability levels attained by different
and usable for the children. In order to evaluate the usability of organizations [3].
the computer game, children must be observed while exploring
and playing the game and given the chance to express their
opinions. Therefore, to involve children effectively, appropriate
evaluation techniques are needed. This paper will discuss an II. USABILITY TESTING
observation method that has been adopted in a usability testing Usability is one of the most important quality
with the children. In this study, a computer game called Jelajah characteristics of a product or system. Usable systems are
is chosen as the game to be tested for the usability evaluation
often easy to learn, efficient, not error-prone and satisfactory
with children. Jelajah is suitable for the experiment because it is
an adventure game which is a very common game genre for in use [4]. Usability brings many benefits, which include
children. Five preschool children aged between five and six years increased productivity, enhanced quality of work, reductions
old are selected for this study. During the usability testing session, in support and training cost, and improved user satisfaction [5].
facilitator observes the child and fill in the observation checklist Usability has not been defined consistently, and various
form in order to track the effectiveness and efficiency of the definitions exist. There is not one agreed upon definition of
game. After the test session completed, the child is required to usability, and usability certainly cannot be expressed in one
answer a simple interview-based questionnaire to know their objective measure. Several researchers have proposed
level of satisfaction after playing the game. Data collected are definitions and categorizations of usability but there is still
both qualitative and quantitative. The collected data are
some consensus on the concept and they mostly differ on
analyzed to determine the levels of effectiveness, efficiency and
satisfaction. The levels of effectiveness, efficiency and satisfaction more detailed levels [6].
are being measured by averaging them to a single metric that can One of the well-known usability definitions is by Jacob
represent the usability of Jelajah. The result obtained showed Nielsen which stated “usability is a quality attribute that
that the approach adopted to conduct a usability testing of a assesses how easy user interfaces are to use, making it
computer game with children using Jelajah can also be adopted possible to the customers to develop tasks in a clear,
for evaluating any educational computer game with children. transparent, agile and useful way” [7]. Nielsen considers that
the usability of a system can have five quality components:
Keywords— Usability; Educational computer game;
Observation method
1) Learnability: How easy is it for the users to
accomplish basic tasks the first time they encounter
I. INTRODUCTION the design?
2) Efficiency: Once users have learned the design, how
Nowadays, computers are not just being used by domain quickly can they perform tasks?
experts such as engineers and programmers. They are now 3) Memorability: When users return to the design after a
used by almost everyone including children who want to get period not using it, how easily can they reestablish
exposed to computers and information technology at an early proficiency?
age. For most children the first contact with the computer is 4) Errors: How many errors do users make, how severe
through some sort of educational game [1]. Therefore, it is are these errors, and how easily can they recover
important that computer games for children are well-designed from the errors?
and should have a good usability. 5) Satisfaction: How pleasant is it to use the design?

The term usability is used to denote that a design is good However the definition from ISO 9241-11 which stated
from Human-Computer Interaction (HCI) point of view and usability as “the extent to which a product can be used by
most efforts by HCI researchers and practitioners are specified users to achieve specified goals with effectiveness,
ultimately aimed at achieving more usable computer systems efficiency and satisfaction in a specified context of use [8]”
[2]. becoming the main reference in usability. This definition
represents a wide scope of usability, emphasizing how the
In general, usability is the extent to which a product or a usability of a product relates to its context of use [9].
system enables users to achieve specified goals. Usability is
also the ability of a product or system to effectively and
efficiently fulfill the needs and specifications of users.
Today, many organizations have now started incorporating

978-1-4244-5651-2/10/$26.00 ©2010 IEEE 157

Authorized licensed use limited to: b-on: UNIVERSIDADE DO MINHO. Downloaded on July 21,2023 at 16:10:37 UTC from IEEE Xplore. Restrictions apply.
 TABLE 1 the usability test, debriefing the participant and analyzing the
USABILITY AS ISO 9241-11 AND JACOB NIELSEN data of the usability test (Figure 1).

ISO 9241-11 Jacob Nielsen

Efficiency Efficiency
Learnability
Effectiveness Memorability
Errors / Safety
Satisfaction Satisfaction

Table 1 shows the definitions mapping between ISO 9241-11

and Nielsen’s as pointed out by [9]. From the table, it was
proved that Nielsen and the ISO standard give a concise
outline of the term usability.
There are two major considerations when conducting
usability testing. The first is to ensure that the best possible
method for testing is used. Generally, the best method is to
conduct a test where representative participants interact with
representative scenarios. The facilitator collects data on the
participant’s success (effectiveness), speed of performance
(efficiency), and satisfaction. The findings, including both
quantitative data and qualitative observations information, are
provided to designers in a test report. Using ‘inspection
evaluations’ in a well-controlled environment, usability tests
must be done with caution.
The second major consideration is to ensure that an iterative
approach is used. After the first test results are provided to the Fig 1. Methodology of usability testing on Jelajah
designers, they should make changes and then have the
product tested again. Generally, the more iteration on usability
testing, the better the product [10]. A. Planning a Usability Test

III. OBSERVATION METHOD Planning process is an essential part of a usability testing.

Observation method is the evaluation method that collects Careful planning will ensure fully understanding and smooth
data by observing users’ experiences with a product [1]. Often, process progress during usability testing. In this step objective,
observational usability evaluations are performed to determine goals and problem statement will be indentified.
quantitative measures like efficiency, effectiveness, and
satisfaction [8]. This method has been proven by many B. Identify Usability Model
researches as one of the best technique that could be used in
usability testing. The approach is to have a session with end In order to identify usability model, we need to set
users, and observe them while performing a task. During dimensions of usability testing, identify the courseware
observation, a representative user is asked to perform several elements and identify methods of usability testing to be used.
authentic tasks using the prototype or instructional resource. 1) Dimensions of usability testing
This study will be based on usability definitions and
A video camera captures what is happening on the users characteristics done by ISO 9241-11 which will
screen, their interaction and navigation of the product, and focus on measuring the effectiveness, efficiency and
also the comments they made when interacting with the level of satisfaction from children.
product. Additional video cameras can be used to record the
users’ facial expressions, body language, keyboard and mouse, 2) Identify the suitable method
or any other important part of the user environment. Upon Observational evaluation has been chosen as the
completion of each test, users will need to answer post method to be used in the usability testing. The
interview like questionnaire about other subjective inputs usability testing will be conducted in a kindergarten.
regarding the product such as ease of use and levels of Each session should not be more than 20 minutes.
satisfaction. The sessions will be facilitated and observed by only
one facilitator.
IV. METHODOLOGY OF THE RESEARCH
There are nine activities involved in the methodology used in 3) Children need to be encouraged to participate and
this research. Those are planning a usability test, identify facilitator should stress the value of the child’s input
usability model, preparing the test, selecting a representative and show appreciation and gratitude. It is extremely
sample, ethical issues in recruiting participants, conducting important to make the child feel at ease[11]. For this

158

Authorized licensed use limited to: b-on: UNIVERSIDADE DO MINHO. Downloaded on July 21,2023 at 16:10:37 UTC from IEEE Xplore. Restrictions apply.
 reason, testing will be done in the kindergarten documents such as parent consent form,
classroom (familiar environment) with the aid of the demographic questionnaire, observation form
teacher. When testing with young children facilitator and checklist, and post test questionnaire.
notes are crucial, rather than relying on verbal
comments, expressions such as smiling or sighing 4) Identify task scenarios
often are used to judge the quality of interaction. Task scenarios for this usability test will be based on
the game’s instructional task and not predefined task
C. Preparing the Test Materials that children need to accomplish since this is
irrelevant to evaluate games. According to [12],
The following is a list of materials that need to be prepared games are entertainment and have to be fun. This has
when running the usability test in this research: nothing to do with accomplishing tasks as quickly or
as efficiently possible.
1) Select test facilitator Hence, our test scenarios are actually the game
Only one facilitator will handle the usability test with instructions which help children to understand on
children in this study. The facilitator will sit next to how the game should work and what they should do.
the child during the session and his/her role is to fill The task scenarios have been divided according to
in the observation form while interacting with the several screens in Jelajah which are; starting screen,
child to make them feel at ease. character selection screen, exploration game screen
and word puzzle game screen.
2) Screen the questionnaire
There are two questionnaires that need to be assessed 5) Observation checklist
which are demographic questionnaire and post task Observation method needs facilitator to record all
questionnaire. The demographic questionnaire shall children action, behaviours and facial expressions
be completed by parents in order to know the child’s while observing children playing the game. In order
backgrounds, family environment, experiences with to guide facilitator, an observation checklist is
computer and what are their ordinary tasks using needed. The checklist should be created based on
computer. functionalities of the game and what suppose to be
Post task questionnaire which is administered as done by players in ideal ways. Every checklist item
an interview will be conducted right after each test should be constructed using positive phrasing.
session with the help from the facilitator. The post
task questionnaires contain questions that are D. Selecting a Representative Sample
supposed to capture subjective input from the child
that cannot be captured using the task-based Effective representative sampling is crucial to collect
questions. This includes; the child’s feeling, fun, ease reliable data during usability test. Not only should test
of use and their satisfaction level towards the game. participants reflect the characteristics of the targeted users of
3) Set the test environment the product or service, but they also must be likely to use it.
The usability testing session will be done in a Only then will their experiences and opinions yield reliable
kindergarten classroom. To prepare for the test data for identifying meaningful improvements [13].
environment, a computer, a camera and the screen The first step in recruiting young participants is sending
recording software is required. This is the minimal information about our study to the correct places. Unlike
setups needed to run the usability test with users. adults, recruiting children for a research study is complicated
Below are summary of items needed for the test because we cannot seek permission directly from the children.
environment: Parents must first trust our organization and the legitimacy of
• Computer: The Jelajah game will run on the our research before they allow their child to participate in a
computer and children will need to control the study.
game using the keyboard hooked to this Usability studies can become expensive when it comes to
computer. designing and selecting users. [14] said,
• Sound devices: Sound devices are needed in
order to play the sound and background music “ The best users come from testing no more than five
of the game. This is believed could add more users and running as many small tests as you can afford”
attraction for children to play the game.
• Video camera: Video camera is needed to Nielsen’s study showed that a group of five users were able to
record the behaviors and expression of the find about 80% of the findings in a system and as more users
child while playing the game. were added, less additional information was found, but more
• Screen recording software: A software called and more money was spent to run tests and to compensate
CamStudio will be used to record screen additional users. The idea behind the assumption is that you
activities during the test. can learn more from a group of five completing multiple tests
• Observation Form: It is crucial to have an than you would on 15 participants completing one test.
observation form during usability test session.
The document shall contain several important

159

Authorized licensed use limited to: b-on: UNIVERSIDADE DO MINHO. Downloaded on July 21,2023 at 16:10:37 UTC from IEEE Xplore. Restrictions apply.
 Conforming to Nielsen’s study; the usability testing is movement on the screen. Children are required to complete
conducted with five children with different background, task scenarios that is the game.
mixed gender and aged five to six years old. Data that are collected are both quantitative and qualitative
data. Data are collected while observing the child performing
E. Ethical Issues in Recruiting Participants the task scenarios. In observing the child performing the task
scenarios, the facilitator must be aware on every aspect that
[15] suggests that, before conducting any usability test we the child might show such as their body language, facial
must do the following: expressions and comments. The facilitator needs to record all
verbal comments and complete the observation checklist.
• Receive signed consent from a parent or guardian
before any child under the age of 18 may participate G. Debriefing the Participant
in a usability test – This ensures that the study meets
ethical guidelines and standards. After completing the usability test session with a child,
• Carefully explain the purpose of the research to facilitator needs to ask the child to answer post task
younger participants to assure that they are not questionnaire. Facilitator will need to take notes on feedback
confused and do not feel manipulated in any way. or comments from the child on how they feel about the game.
• Ensure that child participants are protected at all The children’s answers are later translated into a 3 points
times during the usability test – Parents are allowed Likert scale format according to the Coding Scheme (PeKep)
to wait for their child outside the testing room if they as introduced by [11]. This technique is important to measure
so choose. the satisfaction levels of a product.
• Schedule a break halfway through the test session
and remind participants that they can stop at anytime. H. Analyzing the Data of the Usability Test
• Having videotape constantly recording during the
session offers protection for everyone involved. Both qualitative and quantitative data collected from the test
sessions need to be analyzed further in order to have a better
For this study, parents’ permission is obtained prior to understanding about the usability of a product. Based on the
conducting the usability test with the children. Parents ISO9241-11 definition that defines usability as consisting of
were asked to complete a demographic survey and a three components:
consent form for their child’s participation. Parents were 1) Effectiveness: The accuracy and completeness with
explained that the purpose of the test is to test the which customers achieve specified goals.
software and not the children. Parents were also ensured
that this test will endangered their children in any 2) Efficiency: The accuracy and completeness of goals
situation. achieved in relation to resources.
Building a good relationship with participant is also
3) Satisfaction: Freedom from discomfort and positive
another important issue. A successful test session with
attitudes toward the use of the system.
children or adults are usually ones in which we have been
able to establish a good relationship [15] . [11] conducted It is obvious that effectiveness and efficiency have objective
a series of ‘friendly sessions with her testers prior characteristic, while satisfaction has the subjective
conducting the usability test with them. The ‘friendly characteristic to be measured. Based on this understanding,
session’ consists of activities that are enjoyable to the effectiveness and efficiency metrics are measured using a
children such as sing-a-long and storytelling. Having a simple usability measure introduced by [16] by analyzing
good rapport with test participants puts them at ease, and user’s success rate.
helps them to feel more comfortable sharing their [14] defined success rate as the percentage of tasks that
thoughts and insights than if they do not trust the test users complete correctly. This is a coarse metric. It does not
facilitator. Prior to the test, the facilitator visited to the tell about why users fail or how well they perform the tasks
kindergarten to meet the children and build a friendship they completed. Success rates are easy to collect and measure.
with them. This process assisted facilitator while If users cannot accomplish their target task, all else is
conducting the test as the children already know him. irrelevant. In this study, user success can be derived from the
observation checklist which is used by the facilitator while
F. Conducting the Usability Test performing usability test and the measurement of satisfaction
are taken from post questionnaire with the children. However,
The usability test is conducted in the children’s own the success rate calculation is not discussed here.
familiar environment that is the kindergarten. It is conducted
in a closed-session situation where only the students as the
tester and facilitator were in the room to test the Jelajah game. V. CONCLUSION
During the testing period, the child was allowed to play the Conducting a usability test requires a careful planning.
game with minimal intervention or assistance from the Testing with children as the participant requires extra handling
facilitator. Two cameras were used to capture the whole in terms of human and social interaction because the children
session while a software is used to record the cursor cannot speak for themselves. This paper has laid down a
comprehensive approach to conduct a usability testing with

160

Authorized licensed use limited to: b-on: UNIVERSIDADE DO MINHO. Downloaded on July 21,2023 at 16:10:37 UTC from IEEE Xplore. Restrictions apply.
children as the test participant for an educational game.
Testing an educational game requires evaluation of the
instructional effectiveness, efficiency and satisfaction. The
study conducted with the children concluded that this
approach can be adopted to test a usability of an instructional
application.

REFERENCES
[1] W. Barendregt, M. M. Bekker, and M. Speerstra, “Empirical
evaluation of usability and fun in computer games for children.”, in
Proceedings of Human-Computer Interaction INTERACT-03', IOS
Press, Zürich, Switzerland, 2003. pp. 705-708
[2] J. Löwgren (1995), Perspectives on Usability. Department of
Computer and Information Science, Linköping University. Sweden.
[Online] http://webzone.k3.mah.se/k3jolo/Download/r-95-23.pdf,
[3] W. Barendregt, M.M. Bekker, D.G. Bouwhuis, and E. Baauw, (2006).
“Identifying usability and fun problems in a computer game during
first use and after some practice.”, in International Journal of Human-
Computer Studies, 2006, pp 830-846. vol 64(9),
[4] J. Nielsen, Usability Engineering, Morgan Kaufmann Publishers Inc.,
San Francisco, CA, 1993
[5] ISO 9126-1, Software engineering-Product quality- Part 1: Quality
Model. 2000
[6] A. Seffah, J. Gulliksen and M. C. Desmarais. Human-Centered
Software Engineering - Integrating Usability in the Development
Process, Human-Computer Interaction . New York: Springer-Verlag,
2005
[7] J. Nielsen (2003, August 25). Usability 101: Introduction to
Usability. Alertbox, [Online]
http://www.useit.com/alertbox/20030825.html
[8] ISO 9241-11 , Ergonomic requirements for office work with visual
display terminals (VDTs) – Part 11: Guidance on usability.1998.
[9] M. V. Welie , G.C. van der Veer, and A. Eliëns “Breaking down
Usability.”, In Proceedings of Interact 99, 1999. pp. 613-620
[10] S.J. Koyani, R.W. Bailey RW, and J.R. Nall (2003). Research-based
Web design and usability guidelines.[Online]
www.usability.gov/pdfs/guidelines.html.
[11] Marina Ismail. 2009. A multimedia courseware (BACA) for learning
Bahasa Melayu for preschool using the Vygotsky’s approach. PhD
Thesis. Universiti Kebangsaan Malaysia.
[12] B. Zaman, “Evaluating games with children.”in Proceedings of
Interact 2005 Workshop on Child computer Interaction:
Methodological Research. 2005.
[13] D. Hinderer , “Challenges in participant recruiting for usability
testing,” in Proceedings Professional Communication Conference,
1998. IPCC 98. 1998, pp 23-25
[14] J. Nielsen (2000, March 19). Why You Only Need to Test with 5
Users. Alertbox [Online]
http://www.useit.com/alertbox/20000319.html.
[15] M. Patel and C. Paulsen, “Strategies for Recruiting Children for
Usability Tests” . in Proceedings Usability Professionals
Association, 11th Annual Conference.2002.
[16] J. Nielsen (2001, August 5). First Rule of Usability? Don't Listen to
Users, Alertbox, [Online]
http://www.useit.com/alertbox/20010805.html,

161

Authorized licensed use limited to: b-on: UNIVERSIDADE DO MINHO. Downloaded on July 21,2023 at 16:10:37 UTC from IEEE Xplore. Restrictions apply.