The Method and Metric of User Experience Evaluation: A

Systematic Literature Review

Aulia Inan Nur Harry B. Santoso Panca O. Hadi Putra
Faculty of Computer Science, Faculty of Computer Science, Faculty of Computer Science,
Universitas Indonesia Universitas Indonesia Universitas Indonesia
[email protected] [email protected] [email protected]

ABSTRACT result from the use and/or anticipated use of a system, product or
With the growth of User Experience (UX) research field, researchers service”. So, it can be interpreted that UX involves user’s feeling
have developed various ways to implement UX evaluation method. and thought when interacting with the product. While Alben [5]
These evaluation methods have different practice, evaluated applica- defined experience as what users feeling when they interact with
tion, and type of collected aspect. This article provides a systematic the product, how users understand the ways to use the product
literature review on research papers from 2000 to 2019 related to and if it meets their goals. UX focus on user’s feeling, thought and
UX evaluation, to better understand UX evaluation method and behavior when interacting with the product or system [2]. That
its implementation, what kind of application its applied to, and is why UX has become a more popular research field in HCI than
what type of collected metric. The result of this paper presents usability in recent studies. Usability mainly focused on task and
the most frequently used method is self-reported measurement performance [2, 3]. It has limitation comparing to UX that focused
that evaluates self-reported metric and issue-based metric and the in task-related aspect and non-task related aspect.
least frequently used method is physiological measurement that Measuring user’s feeling while interacting with a product, sys-
evaluates emotion and stress metric. tem or service is a significant challenge in UX evaluation. Measuring
UX using self-reported metric is the traditional method and most
CCS CONCEPTS common method in user experience evaluation method (UXEM)
[7]. This method is highly subjective and thus dependent on user’s
• General and reference → Document types; General literature;
interpretation. These subjective natures make a disadvantage in
• Human-centered computing → Human computer interaction
evaluating UX because they create bias in their result [6, 7]. And
(HCI); HCI design and evaluation methods.
even if the questionnaire could be correctly interpreted by a user,
sometimes the user does not give the answer that is true to their
KEYWORDS
emotions. This also makes the self-reported measurement unable to
User Experience, Measurement, Methods, Systematic Literature acquire the true feeling of the user’ emotional experience [7]. Ob-
Review servational measurement method is another method in UXEM that
ACM Reference Format: focused on objective value. This method observes user behavior
Aulia Inan Nur, Harry B. Santoso, and Panca O. Hadi Putra. 2021. The Method while interacting with a product or system. But this method cannot
and Metric of User Experience Evaluation: A Systematic Literature Review. observe user emotional state [7]. Because of this disadvantage, re-
In 2021 10th International Conference on Software and Computer Applications searcher has begun using modern technologies to help evaluate UX.
(ICSCA 2021), February 23–26, 2021, Kuala Lumpur, Malaysia. ACM, New
The UXEM using modern technologies to measure the physiological
York, NY, USA, 11 pages. https://doi.org/10.1145/3457784.3457832
states of the user is called psychophysiological measures method.
The most common of this method include the use of galvanized
1 INTRODUCTION skin response (GSR) to evaluate stress and arousal through skin
For the last two decades, User Experience (UX) has been a popular activities, electroencephalography (EEG) to detect user emotional
research field in the community of Human Computer Interaction response by evaluating brain activities, electromyogram (EMG) to
(HCI) [1]. In early days, the community in HCI mostly focused detect stress level or muscular activities, eye tracking to measure
in analyzing and evaluating behavioral goals, but Hassenzahl et eye movement and visual attention [6-9].
al. [2] argued that HCI should not only focus on pragmatic aspect With the growth of UX research field, researchers have devel-
(behavioral goals) but also hedonic aspect of the interaction between oped various ways to implement UXEM. These evaluation methods
product and user. As it is mentioned in ISO 9241-110:2020 [4], UX have different practice, evaluated application, and type of collected
defined as: “combination of user ’s perceptions and responses that metric. Arifin et al. [10] reviewed research papers related to UX
Permission to make digital or hard copies of all or part of this work for personal or
component, evaluated application, measured metric, and UX as-
classroom use is granted without fee provided that copies are not made or distributed pect from 2010 to 2017. They categorized the metric based on the
for profit or commercial advantage and that copies bear this notice and the full citation proposed standard metric developed by Albert et al. [17]. But they
on the first page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, only reviewed research papers in augmented reality domain and
to post on servers or to redistribute to lists, requires prior specific permission and/or a the research paper did not review the method used for UX eval-
fee. Request permissions from [email protected]. uation. Zarour et al. [1] reviewed research papers that related to
ICSCA 2021, February 23–26, 2021, Kuala Lumpur, Malaysia
© 2021 Association for Computing Machinery. UX aspect and dimension from 2005 to 2015. Their main purpose
ACM ISBN 978-1-4503-8882-5/21/02. . . $15.00 was to discover the relation between UX aspect and UX dimension.
https://doi.org/10.1145/3457784.3457832

307
ICSCA 2021, February 23–26, 2021, Kuala Lumpur, Malaysia Aulia Nur et al.

Table 1: Exclusion and Inclusion Criteria

Inclusion Criteria Exclusion Criteria

The article is journal or conference paper Literature review paper
The article is published in English Language Full paper not accessible
An empirical study Duplicate paper
Answered all research question Non-IT domain

They did not review the UX evaluation and the evaluated prod- service can be used by specified users to achieve specified goals
uct. Rivero et al. [11] created a systematic mapping study of UX with effectiveness, efficiency, and satisfaction in a specified context
evaluation between 2010 and 2015. The systematic mapping study of use”. From that definition, it can be interpreted that usability
reviewed UXEM and its implementation, the evaluated product, focuses on the functionality of the product and whether or not the
and the type of collected data. Their research paper did not review product has attained certain goals while the user uses a product. ISO
the metric evaluated by UXEM, they only reviewed the type of 9241-110:2020 [4] also mention the definition of UX as “combination
collected data (e.g., qualitative or quantitative data). The research of user’s perceptions and responses that result from the use and/or
method used by Rivero et al. [11] is similar to the research method anticipated use of a system, product or service”. The definition
used by Vermeeren et al. [12] because Rivero et al. [11] research is of UX is different than usability. While usability focuses on the
a continuation from Vermeeren et al. research [12]. goal-related interaction between user and product, UX focuses on
Previous researches have not done a review of implementation user’s response and perception that include emotion, preference,
UXEM and its metric. Arifin et al. [10] reviewed the evaluated and behavior of the user.
product, UX aspect, and metric, but did not review the UXEM. There are various perceptions regarding the connection between
Zarour et al. [1] reviewed UX aspect and UX dimension, but did UX and usability. Følstad et al. [13] present three viewpoints re-
not review the UXEM and evaluated product. Rivero et al. [11] garding the connection between UX and traditional usability. The
reviewed the evaluated product, UXEM, and its implementation, but first viewpoint is “UX encompass usability” which defined UX as
did not review the evaluated metric. This research paper is different a broad concept that contains usability and other concepts. The
from the previous research mentioned above. This paper reviews second viewpoint is “UX complements usability” which defined
UXEM to find the relation between UXEM with the evaluated metric UX as usability that focuses on measuring non-goal related aspects.
and the implementation of UXEM. This research paper focuses on The last viewpoint is “UX is one of several components constituting
UXEM published from 2000 to 2019. The result of this paper is usability” which defined UX as part of the usability concept [13].
expected to create an overview of the development of UXEM. Hassan et al. [14] performed a research that reviews the author’s
perception of UX and usability relationships.
2 METHOD Hassenzahl [15] defines a key element of UX from user’s per-
spectives such as apparent product character and consequence.
To understand the development of UX evaluation, this research
Apparent product character is the product feature perceived by
reviews papers published in Scopus, Science Direct IEEE Xplore,
an individual when they interact with the product. The product
ACM Digital Lib, Emerald insight from 2000 to 2019. The inclusion
character perceived by the user consists of pragmatic attributes
and exclusion criteria used in this study are mentioned in Table 1
and hedonic attributes. Pragmatic attribute focuses on utility (i.e.,
This paper main purpose is to review UXEM and how the method
relevant functionality of product) and usability (i.e., ways to access
is implemented, the evaluated product, and the metric that mea-
this functionality), while hedonic attribute focuses on a user physi-
sured using UXEM. The research questions that were used to collect
ological well-being. Hedonic attributes are divided into stimulation,
relevant research papers are as follows:
identification, and evocation. Consequence is related to apparent
• What User Experience Evaluation Method (UXEM) that was product character as the user assessment of product features. The
used and how the method was implemented? consequence could lead to the user emotional consequence e.g. sat-
• What kind of product that the method is applied to? isfaction and pleasure or behavioral consequence e.g. time spent
• What type of metric that was evaluated by the UXEM? using the product [15].
Keywords are extracted from the research question as the search Mahlke [16] proposes a framework called CUE-model (Compo-
term was used to collect relevant research papers. The keyword nent of User Experience). Based on CUE-model, UX component
that was used for the search term is: (UX OR “User Experience” consist of instrumental qualities, non-instrumental qualities, and
OR “User perception*”) AND (Measurement OR Evaluation) AND emotional reaction. Instrumental qualities focusses on utility (i.e.,
(Metric OR Attribute OR Aspect OR Scale). product functionality that fulfill user requirements) and usability
(i.e., how well users utilize the product). While non-instrumental
3 LITERATURE REVIEW qualities focusses on aesthetic, symbolic value, and motivational
aspect. Aesthetic aspect of non-instrumental qualities related to
3.1 Theories of User Experience the appearance of the product. Symbolic value of non-instrumental
In HCI, the term of usability and UX have a close relationship. qualities is subdivided into two dimension, communicative and asso-
ISO 9241-110:2020 [4] describe usability as “a system, product or ciative aspects. Associative aspect relates to personal memories (i.e.,

308
The Method and Metric of User Experience Evaluation: A Systematic Literature Review ICSCA 2021, February 23–26, 2021, Kuala Lumpur, Malaysia

what the user associate with product attributes) while communica- Self-reported metric is used to measure the user’s perception or
tive aspect relates to individual qualities (i.e., self-expressiveness of opinion of a product, system, or service. The last metric is behav-
the user). Motivational aspect of non-instrumental qualities defined ioral and physiological metric. This metric is used to measure user’s
as product capability to motivate the user. The last component of body behavior or reaction when interacting with the product. There
CUE-model is emotional reaction. Emotional reaction is influenced is a possibility that there is a difference in user’s thought and what
by the perception of instrumental and non-instrumental qualities user’s really feels. Behavior and physiological metric show what
[16]. user’s really feels, that is why this metric is useful to gauge UX in
The model proposed by Hassenzahl [15] and Mahlke [16] is sim- objective perspective.
ilar. Mahlke CUE-model [16] consists of instrumental qualities that Hussain et al. [7] categorized UX metric into 3 types of metric:
are reflected in pragmatic aspect of Hassenzahl’s UX model [15] interaction metric, self-reported metric, and emotion and stress
and non-instrumental qualities that are reflected in hedonic aspect metric. The definition of self-reported metric is similar between
of Hassenzahl’s UX model [15]. Both values affect the consequence Hussain et al. [7] and Albert and Tullis [17], which is to record user’s
of interaction e.g. user’s emotion. From the review of Hassan et perception or opinion of product, system, or service. Interaction
al. [14] and the model proposed by Mahkle [16] and Hassenzahl metric divided into behavior metric and performance metrics. The
[15], it could be interpreted that usability is a subset of UX. Us- definition of performance metric is the same as performance metric
ability included in instrumental qualities (pragmatic attribute) of defined by Albert and Tullis [17], but behavior metric is grouped
UX component focuses on product functionality and goal-related within behavior and physiological metric instead with performance
aspect. While aspects that is not related to the goal of the product metric. Hussain et al. [7] performance metric focus on the value of
(e.g., aesthetic aspect) were included in non-instrumental qualities the user’s capability while using the product for example task time,
(hedonic attribute). task success, or error. While behavior metric focus on the value of
common interaction between a user and product e.g. page/screen,
3.2 User Experience Evaluations Method event such as user click. The definition of emotion and stress metric
(UXEM) is similar to the definition of behavioral and physiological metric
by Albert and Tullis [17]. Hussain et al. [7] did not mention issue-
There are many methods of UX evaluation that have been developed.
based metric, a metric that indicates issue or problem that the user
Hussain et al. [7] categorized UXEM into three categories: (1) Self- encountered.
reported measurement method, (2) Observational measurement
method, and (3) Physiological measurement method. Self-reported
4 RESULT
measurement method is the most common method. In this method,
users report their perspective and experience about the product, sys- This section presents the result of our analysis of the research paper.
tem or service without any need of expert intervention. The second There are 3.231 papers (1.826 papers from Scopus, 284 papers from
measurement method is observational measurement method. In ob- Science Direct, 484 papers from IEEE Xplore, 618 papers from ACM
servational method, there is a need for expert intervention. Unlike Digital Lib, and 19 papers from Emerald insight) that were found
in self-reported measurement method, observational measurement based on the search query. There are 128 papers that fulfilled all
method requires an expert to observe the user while interacting inclusion and exclusion criteria. From 128 papers, only 61 papers
with the product. The last method is physiological measurement will be reviewed. 67 papers were excluded due to it having quality
method. Physiological measurement method is used to measure the value less than 2.5 in quality assessment.
biological information of a user. The biological information reflects This section describes the user UXEM that has been implemented
how the user feel during experiencing the product [7]. in the reviewed paper and what is the evaluated product and evalu-
ated metric. The result of the review is described in Table 2. Table
2 illustrate users experience evaluation method and how to imple-
3.3 User Experience Metric
ment the experiment (data source, the location of the experiment,
Metric is a value that is measured to indicate a certain phenomenon and period of evaluation), evaluated product, and evaluated met-
or thing for example temperature, weight, height, time, and many ric and collected data type. The result of the literature review is
others. In UX domain, metric is also used as an indicator that is described in the next paragraph to answer the research question.
measured to gauge the experience of the user while interacting
with a product, system, or service. With UX metric, the researcher 4.1 What User Experience Evaluation Method
can discover a user’s feeling and experience with the evaluated
product or discover an improvement in the product [17].
(UXEM) That Used and How the Method Is
Albert and Tullis [17] present four types of metric that are used Used?
in UX domain. The first metric is performance metric. This metric UXEM is categorized into three types of method: Self-reported mea-
is used to measure user’s behavior while interacting with a product surement, observational measurement and physiological measure-
or system for example user journey (i.e., user behavior) to reach ment [7]. To improve the result of UX evaluation, few researchers
certain web pages (i.e., user goal). The most common performance usually apply a triangulation method which means that they used
metric is the task success, error, or time to take to reach the goal. more than one method and sometimes from different types of
The second metric is issue-based metric. This metric is used to methods [17]. This study found that the most frequently imple-
measure issues or problems that the user encounters while inter- mented method is self-reported method (95%) with 17 papers that
acting with the product. The third metric is self-reported metric. use self-reported measurement and observational measurement

309
ICSCA 2021, February 23–26, 2021, Kuala Lumpur, Malaysia Aulia Nur et al.

(e.g., [23, 27, 29, 30, 34, 36, 46, 49, 51, 54, 55, 57, 59, 62, 67, 68, 73]), For physiological measurement, the only period of evaluation
4 papers that use self-reported measurement and physiological is during usage. There are 2 papers (e.g., [41, 44]) that did not ex-
measurement (e.g., [18, 22, 41, 52]), and 33 papers that only use plain the location of evaluation for physiological measurement,
self-report measurement (e.g., [19–21, 24–26, 28, 31–33, 35, 37– the location of physiological measurement is only implemented in
40, 42, 45, 47, 48, 50, 53, 60–64, 66, 69–71, 74–77]). The second controlled environment (lab). This restriction is possibly caused
measurement method is observational measurement (37%) and the by the necessary equipment needed to measure bioinformatic data
third measurement method is physiological measurement (14%). of the user. Therefore, researchers cannot implement physiologi-
There are 2 papers that only use observational method (e.g., [43, 56]), cal measurement for long-term period of evaluation or outside of
1 paper that only uses physiological method (e.g., [65]) and 4 papers controlled environment.
that use all measurement methods (e.g., [22, 44, 72, 78]).
Most of the research did not mention the method of sampling 4.2 What Kind of Product the Method Is
used in the experiment. The most frequently used sampling is con- Applied to?
venience sampling with total 10 papers (e.g., [23, 25, 27, 33, 38,
For the evaluated product there are many different products that
45, 47, 59, 75, 76]). The other sampling method is purposive sam-
the evaluated method applied to. Most frequently paper evalu-
pling (5 papers e.g. [28, 44, 56, 61, 63]), random sampling (4 papers
ated product within educational domain (17 papers e.g. [8, 18,
e.g. [35, 39, 40, 61]), quota sampling (1 paper e.g. [26]), snowball
34, 35, 38, 39, 41, 44, 45, 49, 59, 61, 63, 71, 73, 74, 76]), for exam-
sampling (1 paper e.g. [33]), and stratified sampling (1 paper e.g.
ple educational institution website, e-Learning system, Learning
[73]).
Management System (LMS), mobile application for education, edu-
The most frequently period of evaluation for implementing self-
cational desktop game, educational Virtual Reality (VR) system,
reported measurement is after usage. There are 44 papers that im-
VR serious game and student information system. The second
plement self-reported measurement after a user used a product (e.g.,
most frequently paper is product in health domain (15 papers e.g.
[18–20, 22, 26], [28-35], [37, 38, 40–42, 44–46, 49], [51-55], [57, 59],
[21, 24, 25, 27, 30, 32, 36, 37, 50, 51, 55, 64, 67, 75, 77]), for example
[60-63], [67-76], [78]). The second is during usage with 12 papers
exercise game, m-Health application, Health Information System
in total (e.g., [21, 23, 30, 32, 34, 36, 44, 48, 55, 58, 59, 61]). Long term
(HIS), bioinformatic software and Health Information Exchange
period of evaluation is implemented with 6 papers (e.g., [24, 25,
(HIE). There are 6 papers (e.g., [23, 26, 28, 29, 56, 70, 72]) that eval-
27, 33, 36, 64, 77]) in self-reported measurement method. The least
uate products in entertainment domain for example mobile game
frequently implemented period of evaluation for self-reported mea-
application, VR game, and Brain–Computer Interface (BCI) game.
surement is before usage with total 5 papers (e.g., [28, 29, 31, 60, 72]).
For evaluated products in travel and guide domain there are 6
There are 4 papers (e.g., [39, 47, 50, 66]) that did not explain the
papers in total (e.g., [21, 28, 31, 40, 43, 58]). There are 4 papers
period of evaluation for self-reported measurement. Self-reported
(e.g., [20, 47, 53, 62]) that evaluated product within business and
measurement was most frequently implemented in controlled envi-
finance domain, for example digital wallet, cryptocurrency website,
ronment (lab) with 24 papers in total (e.g., [18, 19, 22, 23, 26, 30, 35–
crowdsourcing mobile application, and ERP (Enterprise Resource
37, 42, 48, 49, 51–53, 55, 59, 60, 62, 63, 69, 71, 72, 78]). The next
Planning) system. Product in commercial domain is also evalu-
most frequently implemented is online or automatically with to-
ated with a total of 4 papers (e.g., [46, 48, 52, 57]). In social and
tal 13 papers (e.g., [24, 25, 27, 34, 36, 49, 64, 66, 73–77]). The least
communication domain, there are 2 papers in total (e.g., [19, 21]).
frequently implemented location for self-reported measurement is
6 papers (e.g., [22, 33, 54, 65, 66, 78]) are evaluated as news and
in non-controlled environment (field) with 6 papers in total (e.g.,
book for example online news, e-Book, or e-Reader. There are 2
[20, 33, 45, 46, 54, 68]). There are 15 papers (e.g., [21, 29, 31, 32, 38–
papers (e.g., [31, 68]) that evaluate product in industrial domain.
40, 44, 47, 50, 57, 58, 61, 67, 70]) that did not explain the location of
The least evaluated product is in culinary domain (e.g., [42]) and
self-report measurement.
home management domain (e.g., [60]).
As for the period of evaluation of observational measurement
method, the most frequently period of evaluation for implementing
observational measurement is during usage with total 20 papers
4.3 What Type of Metric That Was Evaluated
(e.g., [22, 23, 29, 30, 34, 43, 44, 46, 49, 51, 54–57, 59, 62, 67, 72, 73, 78]). by the UXEM?
There are 2 papers that implement observational measurement Based on the result, most of the paper that were reviewed is
in long-term period of evaluation (e.g., [27, 36]). For location of collecting metric with quantitative data (60 papers e.g. [18-44],
experiment for observational measurement, there are 13 papers that [46-78]). While, qualitative data is measured in 21 papers (e.g.,
implement observational measurement in controlled environment [19, 21, 32, 33, 35–37, 42, 45, 48, 50, 55, 57–61, 63, 66, 68, 75]). Qual-
(lab) e.g. [22, 23, 30, 49, 51, 55, 56, 59, 62, 68, 72, 73, 78]. The next most itative data consist of issue-based metric and self-reported metric.
frequent, observational measurement is conducted over the internet Though, most of the self-reported metric is categorized into quanti-
(online) with 5 papers in total e.g. [27, 34, 36, 43, 49]. There are 3 tative data. All of Interaction metric and stress and emotion metric
papers (e.g., [46, 54, 68]) implement observational measurement in consist of quantitative data.
non-controlled environment (field), while there are 4 papers (e.g., There are 12 papers (e.g., [21, 23, 32, 36, 37, 42, 47, 57–59, 61, 68])
[29, 44, 57, 67]) that did not explain the location of observational that measure issue-based metric. This evaluated metric represents
measurement method. problems that user encounter during interaction. There are also
paper that measure severity level and the number of usability issue.
9 reviewed papers (e.g., [18, 22, 41, 52, 56, 57, 65, 72, 78]) measure

310
The Method and Metric of User Experience Evaluation: A Systematic Literature Review ICSCA 2021, February 23–26, 2021, Kuala Lumpur, Malaysia

Table 2: The Distribution of User Experience Evaluation, Procedure, Evaluated Product and Evaluated Metric

Ref User Experience Evaluation Method (UXEM) Evaluated Product Evaluated Metric Data
Type
UXEM Categories Theo. Method of Exp. Loc. Eval. SR INT SE IB QN QL
SR OBS PHY Framework Sampling Per. B P
√ √
[18] GSR DU Edu. game
√ √
BVP
√ √
Quest. AU
√ √
[19] Quest. Lab AU Communication
√ √
Interview mobile app. √ √
[20] AttrakDiff Field AU Mobile wallet
SCWS wallet
√ √ √
[21] HE DU Social web. svc.
Travel web. svc.
Health web. svc.
√ √
[22] Smiley-o- Convenience Lab AU e-Book
Meter
√ √
Interview AU
√ √
ET DU
√ √
PE DU
√ √
Ranking AU
survey
√ √
[23] HE Lab DU Travel booking
√ √
PE web. √ √
[24] Quest. Online LT HIS
√ √
[25] QUIS Convenience Online LT HIE
√ √
[26] GEQ Quota Lab AU Mobile game
app.
√ √
[27] Quest. UTAUT Convenience Online LT HIS
√ √
INTR
√ √
[28] Attrakdiff Purposive Lab BU Travel Mobile AR
√ √
SUXES AU app.
√ √
Emocard AU Virtual Pet AR app.
√ √ √
[29] INTR Convenience DU VR game
√ √
ITC-SOPI BU Desktop game
Presence AU √ √
SSQ
√ √
[30] Think Lab DU m-Health app.
aloud
√ √
PE
√ √
Quest. AU
√ √
[31] SUXES BU Travel public
√ √
Quest. Experience AU display app.
Pyramid Industrial public
display app.
√ √
[32] Think DU Bioinformatic
aloud software √ √
SUMI AU
√ √
Interview
√ √
[33] Flow Convenience Field LT e-Book
Short Snowball
Scale
√ √
Interview AU
√ √
[34] ASQ Online AU Edu. web.
√ √
WAMMI
√ √ √
INTR DU

311
ICSCA 2021, February 23–26, 2021, Kuala Lumpur, Malaysia Aulia Nur et al.

√ √
[35] Quest. Random Lab AU BCI serious game
√ √
Quest.
√ √
[36] Think Lab DU HIS
aloud
√ √
Quest. Online LT
√ √
VULab
log
data
√ √
VULab
quest.
√ √
[37] SUS Lab AU Exercise game
√ √
Interview UTAUT software √ √
Ranking
card
√ √
[38] GEQ Convenience AU Edu. games
√ √
IMI
√ √
[39] Checklist Random Edu. web.
√ √
Quest.
√ √
[40] UEQ Random AU TSP app.
√ √
[41] EEG DU Edu. VR
√ √
SAM AU
√ √ √ √
[42] Quest. Lab AU Culinary mobile
app.
Culinary
wearable app.
√ √ √
[43] INTR Online DU Travel online
quiz web.
√ √
[44] Quest. Purposive AU Educational game
√ √
INTR DU web. app. √ √
FEO
√ √
GSR
√ √
BVP
√ √
RESP
√ √
Quest. PAD model
√ √
[45] Interview Convenience Field AU VR serious game
√ √
[46] PE Field DU m-Commerce svc.
√ √
SUS AU
√ √
[47] HE Convenience ERP system
√ √
[48] SUS Lab AU m-Commerce app.
√ √
Think- DU
aloud
√ √ √
[49] INTR Lab DU e-Learning system
√ √
SUS Online AU
√ √
[50] SUS Fall prevention
√ √
PACES system √ √
Quest. DART
√ √
Interview
√ √
[51] SUS Lab AU m-Health app.
√ √
PE DU
√ √
[52] Quest. Lab AU House rent web.
√ √
ET DU
√ √
HRV
√ √
[53] Quest. ISO 25010 Lab AU Crowdsourcing
mobile app.
√ √
[54] Quest. Field AU e-Reader mobile
√ √
PE DU app.

312
The Method and Metric of User Experience Evaluation: A Systematic Literature Review ICSCA 2021, February 23–26, 2021, Kuala Lumpur, Malaysia

√ √
[55] PE Lab DU m-Health app.
√ √
Think
aloud
√ √
Quest. AU
√ √
Emoji
card
√ √
[56] FEO Purposive Lab DU Mobile game
app.
√ √
[57] SAM AU Travel mobile app.
√ √
FEO DU
√ √
Quest. AU
√ √ √
[58] HE DU Museum guide
mobile app.
√ √
[59] SUS Convenience Lab AU e-Learning system
√ √
AttrakDiff
√ √
Interview
√ √
Think DU
aloud
√ √
PE
√ √
[60] MDMQ Lab BU AU Spatial AR app.
√ √
SAM
√ √
Quest. AU
√ √
Quest.
√ √
Quest.
√ √
[61] UEQ Random AU SIsKA web.
√ √ √
HE Purposive DU
√ √
[62] SEQ Lab AU Cryptocurrency
√ √
SUS web. √ √
QUIS
√ √
USE
√ √
Product
reaction
cards
√ √
PE DU
√ √
[63] UEQ Purposive Lab AU LMS
√ √
Interview
√ √
[64] NuHISS Online LT HIS
√ √
[65] EEG Lab DU Online news
article
√ √
[66] UEQ Online e-Learning system
√ √
HE quest.
√ √
[67] SUS ISO 9241-11 AU m-Health app.
√ √
PE DU
√ √
[68] PE Field DU Industrial AR HMD
√ √ √
Interview AU app. √ √
[69] GUESS Lab AU VR game Desktop
√ √
Presence game
Scale
√ √
[70] UEQ AU Entertainment
mobile AR app.
√ √
[71] Quest. Lab AU Edu. VR system
√ √
[72] Quest. Lab BU Entertainment
√ √
PE DU mobile AR app. √ √
EEG
√ √
UEQ AU
√ √
SUDs

313
ICSCA 2021, February 23–26, 2021, Kuala Lumpur, Malaysia Aulia Nur et al.

√ √
[73] SUS Stratified Online AU Student
√ √
EWPL information system√ √
Quest.
√
PE Lab DU
√ √
[74] Quest. Online AU Edu. game
√ √
[75] uMARS Convenience Online AU HIS
√ √
Quest.
√ √
[76] PEDEACLWQ Convenience Online AU Edu. web.
√ √
[77] Quest. Online LT HIS
√ √
[78] UES Lab AU News web.
√ √
CAS
√ √
SUS
√ √ √
INTR DU
√ √
HRV
√ √
EDA
√ √
EMG

User Experience Evaluation Method (UXEM). Theo. Framework: Theoretical Framework; Exp. Loc.: Experiment Location; Eval. Per.:
Evaluation Period.
UXEM Categories. SR: Self-Reported; OBS: Observational; PHY: Physiological.
Eval. Per. AU: After Usage; DU: During Usage; BU: Before Usage; LT: Long Term
Evaluated Metric. SR: Self Report; INT: Interaction; B: Behavior; P: Performance; SE: Stress and Emotion; IB: Issue-Based
Data Type. QN: Quantitative; QL: Qualitative
SR. QUIS: Questionnaire for User Interface Satisfaction; GEQ: Game Experience Questionnaire; SSQ: Simulator Sickness Questionnaire;
SUMI: Software Usability Measurement Inventory; ASQ: After-Scenario Questionnaire; WAMMI: Website Analysis and Measurement
Inventory; SUS: System Usability Scale; IMI: Intrinsic Motivation Inventory; SAM: Self-Assessment Manikin; PACES: Physical Activity
Enjoyment Scale; MDMQ: Multidimensional Mood State Questionnaire; SEQ: Single Ease Question; UEQ: User Experience
Questionnaires; NuHISS: National Usability-focused HIS Scale; GUESS: Game User Experience Satisfaction Scale; SUDs: Subjective units
of distress scale; EWPL: Emotion Words Prompt List; uMARS: Mobile App Rating Scale; PEDEACLWQ: Perceived Design Effectiveness of
Nigerian Academic Library Websites Questionnaire; USE: Usefulness, Satisfaction, and Ease of use Questioner; CAS: Cognitive
Absorption Scale; UES: User Engagement Scale; Quest.: Questionnaire
OBS. PE: Performance evaluation; INTR: Interaction record; FEO: Facial expression observation
PHY. GSR: Galvanic Skin Response; BVP: Blood Volume Pulse; ET: Eye Tracking; EEG: Electroencephalography; RESP: Respiration; HRV:
Heart Rate Variability; EDA: Electrodermal Activity; EMG: Electromyography
Theo. Framework. UTAUT: Unified Theory of Acceptance and Use; DART: The Dynamic Acceptance Model for the Re-evaluation of
Technologies; PAD: Pleasure Arousal Dominance
Evaluated Product. App.: Application; HIS: Health Information System; HIE: Health Information Exchange; AR: Augmented Reality;
VR: Virtual reality; ERP: Enterprise Resource Planning; SIsKA: Academic Progress Information System; LMS: Learning Management
System; TSP: Travelling Salesman Problem; Web.: Website; Edu.: Educational; Svc.: Service; BCI: Brain–Computer Interface

emotion and stress metric. The metric is evaluated from physio- 5 DISCUSSION
logical based emotion recognition to measure bioinformatics (e.g., Self-reported measurement method could be used for measuring
visual attention, heart beats interval, skin conductivity) and video- the problem a user may encounter during interacting with prod-
based emotion recognition to observe emotion from observing uct. The value is categorized into issue-based metric as mentioned
facial expression. Interaction metric is divided into 2 sub metric, by Albert and Tullis [17]. Although most of the issue-based met-
performance metric and behavior metric. There are 18 papers (e.g., ric is categorized as qualitative data, there are some research that
[22, 23, 29, 30, 34, 43, 46, 49, 51, 54, 55, 59, 62, 67, 68, 72, 73, 78]) categorized issue-based metric in quantitative data for measuring
that measure performance metric, while there are 8 papers (e.g., the severity rating of the usability issue (e.g., [42, 47, 58, 61]). The
[27, 29, 34, 36, 43, 44, 49, 78]) that measure behavior metric. 54 method used for evaluating issue-based metric is heuristic evalua-
papers (e.g., [18-20], [22], [24-42], [44-55], [57], [59-64], [66-78]) tion (e.g., [21, 23, 47, 58, 61]), interview (e.g., [32, 37, 68]), think aloud
measure self-reported metric for example pragmatic quality, he- (e.g., [32, 36, 59]) and questionnaire (e.g., [42, 57]). Beside issue-
donic quality aesthetic, usability, enjoyment, emotional response, based metric, Self-reported measurement evaluating self-report
perceived usefulness, learnability, ease of use, etc. metric in form of qualitative data and quantitative data. Method
used for evaluating such metric is think aloud (e.g., [30, 49, 55]),

314
The Method and Metric of User Experience Evaluation: A Systematic Literature Review ICSCA 2021, February 23–26, 2021, Kuala Lumpur, Malaysia

interview (e.g., [19, 22, 33, 45, 50, 53, 63, 68]), non-standard ques- frequently used method is physiological measurement method. Self-
tionnaire (e.g., [18, 19, 22, 24, 27, 30, 31, 35–37, 39, 42, 44, 50, 52– reported measurement and observational method could be imple-
55, 57, 60, 71, 72, 74, 75, 77]) and standard questionnaire (e.g., mented over the internet (online), in controlled environment (lab),
[20, 22, 25, 26, 28, 29], [31-34], [37, 38, 40, 41, 46, 48–50, 55, 57, 59], and non-controlled environment (field) while physiological method
[60-64], [66, 67, 69, 70], [72-76], [78]) . The most used method for is only implemented in controlled environment and only in a cer-
evaluating self-report metric is standard questionnaire. The most tain time period of usage (during usage). The result of this study
common method used for standard questionnaire is System Usabil- may be used as a reference by researchers or students for analysis
ity Scale (e.g., [37, 46, 49–51, 53, 62, 67, 73, 78]) and User Experience regarding UX evaluation, the procedure for implementation, and
Questionnaire (e.g., [40, 61, 63, 70, 72]). Some of the non-standard evaluated metric.
questionnaire is developed based on theory, for example Unified
Theory of Acceptance and Use model (e.g., [27, 37]), Experience ACKNOWLEDGMENTS
Pyramid theoretical (e.g., [31]), Pleasure-Arousal-Dominance emo-
This research was supported by Hibah Publikasi Terindeks Inter-
tion representation model (e.g., [44]), The Dynamic Acceptance
nasional (PUTI) Q2 2020 at Universitas Indonesia (Number: NKB-
Model for the Re-evaluation of Technologies (e.g., [50]), ISO 25010
846/UN2.RST/HKP.05.00/ 2020).
(e.g., [53]), ISO 9241-11 (e.g., [67]).
Observational method is mostly used to measure interaction
method. As previously explained Hussain et al. [7], interaction REFERENCES
metric divided again into performance metric and behavior metric. [1] Mohammad Zarour and Mubarak Alharbi. 2017. User Experience Aspects and
Dimensions: Systematic Literature Review. Cogent Engineering 4, 1 (December
Observational method that mostly used to measure performance 2017), 52-59. https://doi.org/10.1080/23311916.2017.1421006
metric is interaction record and performance evaluation. Interaction [2] Marc Hassenzahl and Noam Tractinsky. 2006. User experience – a research
agenda.Behaviour & Information Technology 25, 2 (March 2006), 91-97. https:
record may assist performance metric measurement that was con- //doi.org/10.1080/01449290500330331
ducted over the internet (e.g., [34, 43, 49]) or in controlled environ- [3] Effie Lai-Chong Law, Virpi Roto, Marc Hassenzahl, Arnold P.O.S. Vermeeren,
ment (e.g., [49, 78]). Performance evaluation mostly implemented in and Joke Kort. 2009. Understanding, scoping and defining user experience: a
survey approach. In Proceedings of the SIGCHI Conference on Human Factors
controlled environment (e.g., [22, 23, 30, 34, 51, 53, 55, 62, 78]) or in in Computing Systems (CHI ’09). Association for Computing Machinery, New
non-controlled environment (e.g., [46, 54, 68, 72, 73]). So, it can be York, NY, USA, 719–728. https://doi.org/10.1145/1518701.1518813
concluded that for performance metric, the most common method [4] ISO 9241-110. 2020. https://www.iso.org/obp/ui/#iso:std:iso:9241:-110:ed-2:v1:en
[5] Lauralee Alben. 1996. Quality of experience: defining the criteria for effective
is performance evaluation implemented in controlled environment. interaction design. interactions 3, 3 (May/June 1996), 11–15. https://doi.org/10.
For behavior metric, observational method that is used is only in- 1145/235008.235010
[6] Manon Holman and Funmi Adebesin. 2019. Taking the Subjectivity out of UX
teraction record. The method is implemented over the internet Evaluation with Emotiv EPOC+. In Proceedings of the South African Insti-
(e.g., [27, 34, 36, 43, 49]) or in controlled environment (e.g., [49, 78]). tute of Computer Scientists and Information Technologists 2019 (SAICSIT ’19).
Other than interaction metric, observational method measure stress Association for Computing Machinery, New York, NY, USA, Article 30, 1–10.
https://doi.org/10.1145/3351108.3351139
and emotional metric such as facial expression observation (e.g., [7] Jamil Hussain, Wajahat Ali Khan, Taeho Hur, Hafiz Syed Muhammad Bilal, Jae-
[56, 57]) hun Bang, Anees Ul Hassan, Muhammad Afzal, and Sungyoung Lee. 2018. A
Physiological method is mostly used for measuring stress and Multimodal Deep Log-Based User Experience (UX) Platform for UX Evaluation.
Sensors 18, 5(18 May 2018), 1622. https://doi.org/10.3390/s18051622
emotion metric. Physiological method could be used to measure [8] Fu Guo, Yi Ding, Weilin Liu, Chang Liu, and Xuefeng Zhang. 2016. Can eye-
eye movement (Eye tracking e.g. [22, 52]) or bioinformatic such as tracking data be measured to assess product design? Visual attention mechanism
should be considered. International Journal of Industrial Ergonomics 53 (May
GSR (galvanic skin response) to measure skin conductivity (e.g., 2016), 229-235. https://doi.org/10.1016/j.ergon.2015.12.001
[18, 44]) , BVP (Blood Volume Pulse) to measure blood pulse (e.g., [9] Camila Loiola Brito Maia and Elizabeth S. Furtado. 2016. A study about psy-
[18, 44]), HRV (Heart Rate Variability) to measure heartbeat interval chophysiological measures in user experience monitoring and evaluation. In
Proceedings of the 15th Brazilian Symposium on Human Factors in Computing
(e.g., [52, 78]), electrodermal activity (EDA) to measure produced Systems (IHC ’16). Association for Computing Machinery, New York, NY, USA,
sweat (e.g., [78]), measure respiration (e.g., [44]), electromyography Article 7, 1–9. https://doi.org/10.1145/3033701.3033708
(EMG) to measure muscle activity (e.g., [78]), and EEG to measure [10] Yulyani Arifin, Thomas Galih Sastria, and Edo Barlian.2018. User Experience
Metric for Augmented Reality Application: A Review, Procedia Computer Science
brain activity (e.g., [41, 65, 72]). By using physiological information, 135 (January 2018), 648-656. https://doi.org/10.1016/j.procs.2018.08.221"
the researcher can conclude the emotion that the user feels during [11] Luis Rivero and Tayana Conte. 2017. A Systematic Mapping Study on Re-
search Contributions on UX Evaluation Technologies. In Proceedings of the
interaction with the product for example joy, fear, anger, frustration, XVI Brazilian Symposium on Human Factors in Computing Systems (IHC 2017).
etc. Few papers implement physiological metric even though this Association for Computing Machinery, New York, NY, USA, Article 5, 1–10.
specific method is appropriate for evaluating user’s emotion in https://doi.org/10.1145/3160504.3160512
[12] Arnold P. O. S. Vermeeren, Effie Lai-Chong Law, Virpi Roto, Marianna Obrist,
objective perspective. This measurement method could also only Jettie Hoonhout, and Kaisa Väänänen-Vainio-Mattila. 2010. User experience
be implemented in a controlled environment due to the need for evaluation methods: current state and development needs. In Proceedings of the
certain equipment. 6th Nordic Conference on Human-Computer Interaction: Extending Boundaries
(NordiCHI ’10). Association for Computing Machinery, New York, NY, USA,
521–530. https://doi.org/10.1145/1868914.1868973
[13] Asbjørn Følstad and Rolf Kenneth Rolfsen. 2006. Measuring the Effect of User
Experience Design Changes in e-Commerce Web Sites: a Case on Customer Guid-
ance. In Proceedings of the 2nd International COST294-MAUSE Open Workshop.
6 CONCLUSION COST, Oslo, Norway, 10-15.
This research paper presents a systematic literature review of the [14] Hala Magdy Hassan and Galal Hassan Galal-Edeen. 2017. From usability to
user experience. In Proceedings of the International Conference on Intelligent
UX evaluation research. The review has found that the most fre- Informatics and Biomedical Sciences (ICIIBMS). IEEE, Okinawa, Japan, 216-222.
quently used method in UXEM is self-reported metric and the least https://doi.org/10.1109/ICIIBMS.2017.8279761.

315
ICSCA 2021, February 23–26, 2021, Kuala Lumpur, Malaysia Aulia Nur et al.

[15] Marc Hassenzahl. 2003. The Thing and I: Understanding the Relationship Between 2014 conference on Interaction design and children (IDC ’14). Association for
User and Product. In: Blythe M.A., Overbeeke K., Monk A.F., Wright P.C. (eds) Computing Machinery, New York, NY, USA, 135–144. https://doi.org/10.1145/
Funology. Human-Computer Interaction Series, vol 3. Springer, Dordrecht. https: 2593968.2593978
//doi.org/10.1007/1-4020-2967-5_4 [34] Sharmistha Roy, Prasant Kumar Pattnaik, and Rajib Mall. 2014. A quantitative
[16] Sascha Mahlke. 2008. User Experience of Interaction with Technical Systems. approach to evaluate usability of academic websites based on human perception.
PhD Thesis. Egyptian Informatics Journal 15, 3, 159-167. https://doi.org/10.1016/j.eij.2014.08.
[17] William Albert and Thomas Tullis. 2013. Measuring the User Experience: Col- 002.
lecting, Analyzing, and Presenting Usability Metrics. Morgan Kaufmann. https: [35] Fotis Liarokapis, Kurt Debattista, Athanasios Vourvopoulos, Panagiotis Petridis,
//doi.org/10.1016/C2011-0-00016-9 and Alina Ene. Comparing interaction techniques for serious games through
[18] Helmut Prendinger, Junichiro Mori, and Mitsuru Ishizuka. 2005. Using human brain–computer interfaces: A user perception evaluation study. Entertainment
physiology to evaluate subtle expressivity of a virtual quizmaster in a math- Computing 5, 4, 391-399. https://doi.org/10.1016/j.entcom.2014.10.004.
ematical game. Int. J. Hum. Comput. Stud. 62, 2 (February 2005), 231–245. [36] A Kushniruk, J Kaipio, M Nieminen, H Hyppönen, T Lääveri, C Nohr, A M
https://doi.org/10.1016/j.ijhcs.2004.11.009 Kanstrup, M Berg Christiansen, M-H Kuo, and E Borycki. 2014. Human Factors in
[19] Natasa Milic-Frayling, Martin Hicks, Rachel Jones, and Jamie Costello. 2007. On the Large: Experiences from Denmark, Finland and Canada in Moving Towards
the design and evaluation of web augmented mobile applications. In Proceedings Regional and National Evaluations of Health Information System Usability. Con-
of the 9th international conference on Human computer interaction with mobile tribution of the IMIA Human Factors Working Group. Yearb Med Inform 9, 1 (15
devices and services (MobileHCI ’07). Association for Computing Machinery, August 2014), 67-81. https://doi.org/10.15265/IY-2014-0023
New York, NY, USA, 226–233. https://doi.org/10.1145/1377999.1378011 [37] Ather Nawaz, Nina Skjæret, Kristine Ystmark, Jorunn L. Helbostad, Beatrix
[20] Gerald Madlmayr, Dominik Brandlberger, Josef Langer, and Josef Scharinger. 2008. Vereijken, and Dag Svanæs. 2014. Assessing seniors’ user experience (UX)
Evaluation of SmartCard webserver as an application platform from a user’s of exergames for balance training. In Proceedings of the 8th Nordic Confer-
perspective. In Proceedings of the 6th International Conference on Advances in ence on Human-Computer Interaction: Fun, Fast, Foundational (NordiCHI ’14).
Mobile Computing and Multimedia (MoMM ’08). Association for Computing Ma- Association for Computing Machinery, New York, NY, USA, 578–587. https:
chinery, New York, NY, USA, 360–363. https://doi.org/10.1145/1497185.1497263 //doi.org/10.1145/2639189.2639235
[21] Kaisa Väänänen-Vainio-Mattila and Minna Wäljas. 2009. Developing an expert [38] Luis Gustavo Rotoly de Lima, André de Lima Salgado, and André Pimenta Freire.
evaluation method for user eXperience of cross-platform web services. In Pro- 2015. Evaluation of the user experience and intrinsic motivation with educational
ceedings of the 13th International MindTrek Conference: Everyday Life in the and mainstream digital games. In Proceedings of the Latin American Confer-
Ubiquitous Era (MindTrek ’09). Association for Computing Machinery, New York, ence on Human Computer Interaction (CLIHC ’15). Association for Computing
NY, USA, 162–169. https://doi.org/10.1145/1621841.1621871 Machinery, New York, NY, USA, Article 11, 1–7. https://doi.org/10.1145/2824893.
[22] Areej Al-Wabil, Ebtisam Alabdulqader, Latifa Al-Abdulkarim, and Nora Al- 2824904
Twairesh. 2010. Measuring the user experience of digital books with children: [39] Ankur Pant. 2015. Usability evaluation of an academic library website: Experience
An eyetracking study of interaction with digital libraries. In Proceedings of the with the Central Science Library, University of Delhi. The Electronic Library 33,
2010 International Conference for Internet Technology and Secured Transactions. 5, 896-915. https://doi.org/10.1108/EL-04-2014-0067
IEEE, London, 1-7. [40] Nova Eka Diana and Ocky Aditia Saputra. 2015. Measuring user experience of
[23] H Idyawati, Murni Mahmud, and Abu Osman Md Tap. 2010. User Experience: a potential shipment tracking application. In Proceedings of the International
Assessing the effectiveness of Internet booking service. In Proceeding of the 2010 HCI and UX Conference in Indonesia (CHIuXiD ’15). Association for Computing
International Conference on User Science and Engineering (i-USEr). IEEE, Shah Machinery, New York, NY, USA, 47–51. https://doi.org/10.1145/2742032.2742039
Alam, Malaysia, 1-6. https://doi.org/10.1109/IUSER.2010.5716712 [41] Carofiglio, Valeria and Ricci, Giuseppe and Abbattista, Fabio. 2015. User brain-
[24] Johanna Viitanen, Hannele Hyppönen, Tinja Lääveri, Jukka Vänskä, Jarmo Repo- driven evaluation of an educational 3D virtual environment. In Proceedings of
nen, and Ilkka Winblad, 2011. National questionnaire study on clinical ICT sys- 2015 10th Iberian Conference on Information Systems and Technologies (CISTI
tems proofs: Physicians suffer from poor usability. International Journal of Medi- 2015). IEEE, Aveiro, Portugal, 1-7. https://doi.org/10.1109/CISTI.2015.7170553
cal Informatics, 80, 10, 708-725. https://doi.org/10.1016/j.ijmedinf.2011.06.010. [42] Marcin Wichrowski, Danijel Koržinek, and Krzysztof Szklanny. 2015. Google
[25] Cynthia S Gadd, Yun-Xian Ho, Cather Marie Cala, Dana Blakemore, Qingxia Glass Development in Practice: UX Design Sprint Workshops. In Proceedings of
Chen, Mark E Frisse, and Kevin B Johnson. 2011. User perspectives on the usability the Mulitimedia, Interaction, Design and Innnovation (MIDI ’15). Association for
of a regional health information exchange. Journal of the American Medical In- Computing Machinery, New York, NY, USA, Article 11, 1–12. https://doi.org/10.
formatics Association, 18, 5 (27 May 2011), 711-6. https://doi.org/10.1136/amiajnl- 1145/2814464.2814475
2011-000281 [43] Nouzha Harrati, Imed Bouchrika, Abdelkame Tari and Ammar Ladjailia. 2015.
[26] Stephan Engl, and Lennart E. Nacke. 2013. Contextual influences on mobile player Automating the evaluation of usability remotely for web applications via a model-
experience – A game user experience model. Entertainment Computing, 4, 1, based approach. In Proceedings of the First International Conference on New
83-91. https://doi.org/10.1016/j.entcom.2012.06.001." Technologies of Information and Communication (NTIC). IEEE, Mila, Algeria,
[27] Annemie Heselmans, Bert Aertgeerts, Peter Donceel, Siegfried Geens, Stijn Van 1-6. https://doi.org/10.1109/NTIC.2015.7368757
de Velde, and Dirk Ramaekers. 2012. Family Physicians’ Perceptions and Use of [44] Jakub Miler, and Agnieszka Landowska. Designing effective educational games
Electronic Clinical Decision Support During the First Year of Implementation. J - a case study of a project management game. 2016. In Proceedings of the 2016
Med Syst 36, 3677-3684. https://doi.org/10.1007/s10916-012-9841-3 Federated Conference on Computer Science and Information Systems (FedCSIS).
[28] A. Dhir and Mohammed Al-kahtani. 2013. A case study on user experience (UX) IEEE, Gdańsk, Poland, 1657-1661. https://doi.org/10.15439/2016F434"
evaluation of mobile augmented reality prototypes. J. Univers. Comput. Sci. 19, [45] Bruno Fanini and Alfonsina Pagano. 2015. Interface Design for Serious Game
1175-1196. https://doi.org/10.3217/jucs-019-08-1175 Visual Strategies-The Case Study of "Imago Bononiae". In Proceedings of the
[29] Jean-Luc Lugrin, Marc Cavazza, Fred Charles, Marc Le Renard, Jonathan Freeman, 2015 Digital Heritage. IEEE, Granada, Spain, 623-626. https://doi.org/10.1109/
and Jane Lessiter. 2013. Immersive FPS games: user experience and performance. DigitalHeritage.2015.7419583
In Proceedings of the 2013 ACM international workshop on Immersive media [46] Pride Bongiwe Nyambi and Mamello Thinyane Telkom. 2014. Persona mapping
experiences (ImmersiveMe ’13). Association for Computing Machinery, New for usability of ICTD services. In Proceedings of the 2014 IEEE 6th International
York, NY, USA, 7–12. https://doi.org/10.1145/2512142.2512146 Conference on Adaptive Science & Technology (ICAST). IEEE, Ota, Nigeria, 1-7.
[30] Jia Tan, Kari Rönkkö, and Cigdem Gencel. 2013. A Framework for Software https://doi.org/10.1109/ICASTECH.2014.7068128
Usability and User Experience Measurement in Mobile Industry. In Proceedings [47] Joelma Choma, Luciana A.M. Zaina, and Tiago Silva da Silva. 2016. Working
of the 2013 Joint Conference of the 23nd International Workshop on Software beyond technical aspects: an approach for driving the usability inspection adding
Measurement (IWSM) and the 8th International Conference on Software Process the perspective of user experience. In Proceedings of the 34th ACM International
and Product Measurement (IWSM-MENSURA ’13). IEEE Computer Society, USA, Conference on the Design of Communication (SIGDOC ’16). Association for
156–164. https://doi.org/10.1109/IWSM-Mensura.2013.31 Computing Machinery, New York, NY, USA, Article 11, 1–10. https://doi.org/10.
[31] Tuuli Keskinen, Jaakko Hakulinen, Tomi Heimonen, Markku Turunen, Sumita 1145/2987592.2987607
Sharma, Toni Miettinen, and Matti Luhtala. 2013. Evaluating the experiential [48] Yucheng Jin, Joris Klerkx, and Katrien Verbert. 2016. CircleBuy: a visual search
user experience of public display applications in the wild. In Proceedings of the based second screen application of buying products in videos. In Proceedings of
12th International Conference on Mobile and Ubiquitous Multimedia (<i>MUM the 8th ACM SIGCHI Symposium on Engineering Interactive Computing Systems
’13). Association for Computing Machinery, New York, NY, USA, Article 7, 1–10. (EICS ’16). Association for Computing Machinery, New York, NY, USA, 287–292.
https://doi.org/10.1145/2541831.2541840 https://doi.org/10.1145/2933242.2935871
[32] Casper Shyr, Andre Kushniruk, and Wyeth W Wasserman. 2014. Usability study [49] Nouzha Harrati, Imed Bouchrika, Abdelkamel Tari, and Ammar Ladjailia. 2016.
of clinical exome analysis software: Top lessons learned and recommendations. J Exploring user satisfaction for e-learning systems via usage-based metrics and
Biomed Inform 51, (October 2014), 129-136. https://doi.org/10.1016/j.jbi.2014.05. system usability scale analysis. Comput. Hum. Behav. 61, C (August 2016), 463–
004 471. https://doi.org/10.1016/j.chb.2016.03.051
[33] Luca Colombo and Monica Landoni. 2014. A diary study of children’s user ex-
perience with EBooks using flow theory as framework. In Proceedings of the

316
The Method and Metric of User Experience Evaluation: A Systematic Literature Review ICSCA 2021, February 23–26, 2021, Kuala Lumpur, Malaysia

[50] Daryoush D. Vaziri,Konstantin Aal, Corinna Ogonowski, Thomas Von Rekowski, Using Metacognitive Training System at The First Time?. In Proceedings of The
Michael Kroll, Hannah R. Marston, Rakel Poveda, Yves J. Gschwind, Kim Del- 2018 International Conference on Advanced Computer Science and Information
baere, Rainer Wieching, and Volker Wulf. 2016.Exploring user experience and Systems (ICACSIS). IEEE, Yogyakarta, Indonesia, 213-218. https://doi.org/10.1109/
technology acceptance for a fall prevention system: results from a randomized ICACSIS.2018.8618171
clinical trial and a living lab. Eur Rev Aging Phys Act. 13, 6 (10 Juny 2016). [64] Johanna Kaipio, Anne Kuusisto, Hannele Hyppönen, Tarja Heponiemi, and Tinja
https://doi.org/10.1186/s11556-016-0165-z Lääveri. 2019. Physicians’ and nurses’ experiences on EHR usability: Comparison
[51] Noraini Ibrahim, Masitah Ghazali, Tan Wee Quan, Rooster Tumeng, and Shahliza between the professional groups by employment sector and system brand. Inter-
Abd Halim. 2017. An evaluation study on Dengue-Entomological Surveillance national Journal of Medical Informatics 134. https://doi.org/10.1016/j.ijmedinf.
system using alpha acceptance test. International Journal on Advanced Science, 2019.104018.
Engineering and Information Technology, 7, 4-2. 10.18517/ijaseit.7.4-2.3410 [65] Manon Holman and Funmi Adebesin. 2019. Taking the Subjectivity out of UX
[52] Qing-Xing Qu, Fu Guo, and Vincent G. Duffy. 2017. Effective use of human Evaluation with Emotiv EPOC+. In Proceedings of the South African Insti-
physiological metrics to evaluate website usability: An empirical investigation tute of Computer Scientists and Information Technologists 2019 (SAICSIT ’19).
from China. Aslib Journal of Information Management, 69, 4, 370-388. https: Association for Computing Machinery, New York, NY, USA, Article 30, 1–10.
//doi.org/10.1108/AJIM-09-2016-0155 https://doi.org/10.1145/3351108.3351139
[53] Amorim, Ana and Boechat, Glaucya and Novais, Renato and Vieira, Vaninha [66] Siti Nur Aisyiah Syahrir and Sfenrianto. 2019. User experience questioner and
and Villela, Karina. 2017. Quality Attributes Analysis in a Crowdsourcing based heuristics evaluation in online learning environment. Journal of Theoretical and
emergency. In Proceedings of the 19th International Conference on Enterprise Applied Information Technology 97, 4 (February 2019), 1071-1081.
Information Systems - Volume 2: ICEIS. SciTePress, Porto, Portugal, 501-509. [67] Abdul Samad Dahri and Ahmaed Al-Athwari and A. Hussain. 2019. Usability
https://doi.org/10.5220/0006360405010509 evaluation of mobile health application from AI perspective in rural areas of
[54] Azham Hussain, Emmanuel O. C. Mkpojiogu, Ja’afaru Musa, and Salah Mortada. Pakistan. Int. J. Interact. Mob. Technol. 13, 11 (2019), 213-225. https://doi.org/10.
2017. A user experience evaluation of Amazon Kindle mobile application. In 3991/ijim.v13i11.11513
Proceedings of the AIP Conference Proceedings. AIP, 1 (October 2017). https: [68] Andrew Pringle, Stefanie Hutka, Jesse Mom, Robin van Esch, Niall Heffernan,
//doi.org/10.1063/1.5005393 and Paul Chen. 2019. Ethnographic study of a commercially available augmented
[55] Azham Hussain, Emmanuel O. C. Mkpojiogu, Najdawati Mohd Fadzil, and reality HMD app for industry work instruction. In Proceedings of the 12th
Norhasizasuriati Mohd Hassan. 2017. The UX of amila pregnancy on mobile ACM International Conference on PErvasive Technologies Related to Assistive
device. In Proceedings of the AIP Conference Proceedings. AIP, 1 (October 2017). Environments (PETRA ’19). Association for Computing Machinery, New York,
https://doi.org/10.1063/1.5005394 NY, USA, 389–397. https://doi.org/10.1145/3316782.3322752
[56] Kazi Md. Munim, Iyolita Islam, Mahmuda Khatun, Md Mahboob Karim, and [69] Michael Carroll, Ethan Osborne, and Caglar Yildirim. 2019. Effects of VR gaming
Muhammad Nazrul Islam. 2017. Towards developing a tool for UX evaluation and game genre on player experience. In Proceedings of the 2019 IEEE Games,
using facial expression. In Proceedings of The 2017 3rd International Conference Entertainment, Media Conference (GEM). IEEE, New Haven, USA, 1-6. https:
on Electrical Information and Communication Technology (EICT). IEEE, Khulna, //doi.org/10.1109/GEM.2019.8811554
Bangladesh, 1-6. https://doi.org/10.1109/EICT.2017.8275227 [70] Davidavičienė, V., Raudeliūnienė, J. and Viršilaitė, R. 2019. User experience eval-
[57] Giulia de Andrade Cardieri and Luciana Martinez Zaina. 2018. Analyzing User uation and creativity stimulation with augmented reality mobile applications.
Experience in Mobile Web, Native and Progressive Web Applications: A User and Creativity Studies. 12, 1 (Mar. 2019), 34-48. https://doi.org/10.3846/cs.2019.3576.
HCI Specialist Perspectives. In Proceedings of the 17th Brazilian Symposium on [71] Chen-Wei Chang, Shih-Ching Yeh, Mengtong Li, and Eason Yao. 2019. The In-
Human Factors in Computing Systems (IHC 2018). Association for Computing troduction of a Novel Virtual Reality Training System for Gynecology Learn-
Machinery, New York, NY, USA, Article 9, 1–11. https://doi.org/10.1145/3274192. ing and Its User Experience Research. IEEE Access, 7, 1 (2019), 43637 - 43653.
3274201 https://doi.org/10.1109/ACCESS.2019.2905143
[58] Mohd Kamal Othman, Muhd Nur Shaful Sulaiman, Shaziti Aman, and Vesna [72] Fahad Ahmed Satti, Jamil Hussain, H. S. Bilal, W. Khan, A. Khattak, Ju Eun
Popovic. 2018. Heuristic Evaluation: Comparing Generic and Specific Usability Yeon, and S. Lee. 2019. Holistic User eXperience in Mobile Augmented Reality
Heuristics for Identification of Usability Problems in a Living Museum Mobile Using User eXperience Measurement Index. In Proceedings of The 2019 Confer-
Guide App. Adv. in Hum.-Comp. Int. 2018 (2018). https://doi.org/10.1155/2018/ ence on Next Generation Computing Applications (NextComp). IEEE, Mauritius,
1518682 Mauritius, 1-6. https://doi.org/10.1109/NEXTCOMP.2019.8883528
[59] Joachim Majors, Anette Bengs, Sören Granlund, Anu Ylitalo, and Maria Byholm. [73] Denizhan Demirkol and Cagla Seneler. 2019. Evaluation of Student Information
2018. Moodle Moods? A User Experience Study of a Small Private Online Course System (SIS) In Terms of User Emotion, Performance and Perceived Usability: A
for Higher Teacher Education. In Proceedings of the 22nd International Academic Turkish University Case (An Empirical Study). Procedia Computer Science 158
Mindtrek Conference (Mindtrek ’18). Association for Computing Machinery, New (2019), 1033-1051. https://doi.org/10.1016/j.procs.2019.09.145.
York, NY, USA, 228–235. https://doi.org/10.1145/3275116.3275146 [74] Jonathan Moizer, Jonathan Lean, Elena Dell’Aquila, Paul Walsh, Alphonsus (Alfie)
[60] Lisa Schuhmacher, Anne Pagenkopf, Ragavendra Lingamaneni, and Jürgen Keary, Deirdre O’Byrne, Andrea Di Ferdinando, Orazio Miglino, Ralf Friedrich,
Scheible. 2018. Emotion Enhancement through Ubiquitous Media Technology in a Roberta Asperges, and Luigia Simona Sica. 2019. An approach to evaluating the
Smart Kitchen Environment. In Proceedings of the 17th International Conference user experience of serious games. Computers & Education 136 (2019), 141-151.
on Mobile and Ubiquitous Multimedia (MUM 2018). Association for Comput- https://doi.org/10.1016/j.compedu.2019.04.006
ing Machinery, New York, NY, USA, 317–325. https://doi.org/10.1145/3282894. [75] Najeebullah Soomro, Meraj Chhaya, Mariam Soomro, Naukhez Asif, Emily Saur-
3282904 man, David Lyle, and Ross Sanders. 2019. Design, Development, and Evaluation
[61] A.A. Istri Ita Paramitha, Gede Rasben Dantes, and Gede Indrawan. 2018. The of an Injury Surveillance App for Cricket: Protocol and Qualitative Study. JMIR
evaluation of web based academic progress information system using heuristic Mhealth Uhealth 7, 1 (22 January 2019). https://doi.org/10.2196/10978
evaluation and user experience questionnaire (UEQ). In Proceedings of The 2018 [76] Fasola P. Abifarin, Shaka Apodoghe Imavah, and Ayodele S. Olobashola. 2019.
Third International Conference on Informatics and Computing (ICIC). IEEE, Design effectiveness of academic library web sites A comparison of university,
Palembang, Indonesia, 1-6. https://doi.org/10.1109/IAC.2018.8780430 polytechnic, and college sites in Nigeria. The Electronic Library 37, 3 (Juny 2019),
[62] Bagus Anugrah Ramadhan and Billy Muhamad Iqbal. 2018. User Experience 577-591. https://doi.org/10.1108/EL-08-2018-0159
Evaluation on the Cryptocurrency Website by Trust Aspect. In Proceedings of [77] Ryu Hyeongju and Kim Jeongeun. 2019. Evaluation of User Experience of New
2018 International Conference on Intelligent Informatics and Biomedical Sciences Defense Medical Information System. Healthcare Informatics Research 25, 2(April
(ICIIBMS). IEEE, Bangkok, Thailand, 274-279. https://doi.org/10.1109/ICIIBMS. 2019), 73-81. https://doi.org/10.4258/hir.2019.25.2.73
2018.8550019 [78] H. O’Brien and M. Lebow. 2013. Mixed-methods approach to measuring user
[63] Wahyu Nur Hidayat, Setiadi Cahyono Putro, Mukhamad Angga Gumilang, and experience in online news interactions. J. Assoc. Inf. Sci. Technol. 64, 1543-1556.
Indriana Hidayah. 2018. What is Informatics Education Students Impression of https://doi.org/10.1002/asi.22871

317