In: Proc. of the Intern. Workshop on Meaningful Measures: Valid Useful User Experience Measurement (VUUM),.

Reykjavik,
Iceland, 18 June 2008, E.L-C. Law, N. Bevan, G. Christou, M. Springett, and M. Lárusdóttir (pp. 72-78)

Comparing UX Measurements, a case study

Arnold P.O.S. Vermeeren Anita H.M. Cremers
TU Delft, Industrial Design Engineering TNO Human Factors
Landbergstraat 15, Kampweg 5, P.O. Box 23
2628 CE Delft, The Netherlands 3769 ZG Soesterberg, The Netherlands
E: [email protected],T: +31(0)152784218 E: [email protected], T: +31(0)346356310

Joke Kort Jenneke Fokker

TNO Information and Communication Technology TU Delft, Industrial Design Engineering
Eemsgolaan 3, P.O. Box 1416 Landbergstraat 15
9701 BK Groningen, The Netherlands 2628 CE Delft, The Netherlands
E: [email protected], T: +31(0)505857751 E: [email protected], T: +31(0)152789677
ABSTRACT integrated part of a user’s everyday life create a situation in
In this paper we present our preliminary findings of an which the success of a product in terms of value for the user
informal comparison of different types of User eXperience can no longer be fully understood based on mere usability
(UX) measurements and methods during the field trial of a (e.g. efficiency, effectiveness, satisfaction, learnability,
peer-to-peer file sharing application called Tribler. etc.). It becomes more apparent that the nature of user
activity is opportunistic and situated, based on the relation
Author Keywords between individual predispositions (personality,
User experience (UX), usability, framework, measurement, norms/values, emotions/moods, goals and preferences,
methods, requirements, peer-to-peer file sharing. earlier experiences/knowledge, etc), product interaction and
context (social, physical as well as virtual) [4].Furthermore
ACM Classification Keywords ICT increasingly influences our everyday experiences and
H5.m. Information interfaces and presentation (e.g., HCI): emotions, which is often referred to as the UX [4, 5]. In [3]
Miscellaneous. we presented a tentative framework for UX addressing the
changes mentioned above. This framework draws heavily
1 INTRODUCTION on research by Wright & McCarthy [4] and by Desmet &
Within the TUMCAT1 project, a generic, in-situ Testbed for Hekkert [6], as well as on our own previous research [1].
UX measurements of Mobile, Context-Aware applicaTions The framework is meant to provide an overview of all
is being developed. An informal comparison was made of possible aspects of UX. Further, it should help to identify
TUMCAT findings from a first Tribler field trial, a (1) possible important product (interaction) aspects, such as
laboratory usability test and an internet survey [2, 3]. In this design and context features that may play a role in the UX,
paper we report some preliminary findings of a second and (2) suitable evaluation methods and instruments (see
comparative study. In this second Tribler field trial three section 3) for measuring resulting emotions and
studies (a longitudinal study with TUMCAT tools, an experiences, at appropriate moments before, during and/or
expert review and a laboratory test) are cross-validated on after product interaction.
their ability to measure different User eXperience (UX)
aspects. The measurements within the three studies were set Design elements (outer circle of Figure 1) are the product
up to answer UX research questions based on the UX features a designer can manipulate, such as form, colour
framework as described in [3]. Below, the UX framework (providing aesthetics), interaction flow or choice of specific
and different TUMCAT measurement tools are explained in application features/functionality (providing compositional
more detail, followed by a description of the field trial, the experiences and meaning).
results and some conclusions and future work. Aesthetic design aspects (middle circle) relate to a product’s
capacity to delight one or more of our sensory modalities
2 UX FRAMEWORK [6]. These aspects are closely related to design elements
Shifts in focus from functional aspects of Information and such as look, feel, sound, colour, form and their specific
Communication Technology (ICT) towards ICT as an composition. Aesthetic aspects hardly involve cognitive
processing and lead to emotions (inner circle) such as thrill,
fear, excitement, unease, awkwardness, the perception of
1
For more information on TUMCAT see: speed, time and its boundaries [4, 6].
http://www.freeband.nl/project.cfm?id=1126&language=en
 ourselves and others, finding new possibilities and meaning
in them. In some sense-making processes such as
connecting, cognitive processing is hardly involved and
experiences are a direct result of the perception or sensation
of the aesthetics of the product [7]. In other sense-making
processes cognition plays an important role in making sense
of the product’s workings (compositional structure) and its
attributed meaning (e.g. in anticipation, interpreting,
reflecting, appropriating and recounting). The sense-making
processes described here can all happen in parallel or
successively [4].

3 FROM UX FRAMEWORK TO MEASUREMENTS

In earlier work we identified requirements for new UX

• Measure qualitative as well as quantitative and

measurement tools; such tools should [1]:

subjective as well as objective data related to the UX,

as unobtrusively as possible. For example, they should
Figure 1. UX framework capture users’ behaviour (e.g. application feature
usage) and users’ opinions and emotions about the
Compositional design aspects (middle circle) are closely application’s features and relate both kinds of data to
related to usability, pragmatic and behavioural understand the results in the context in which the
characteristics of an interactive product. They encompass
• Support long term studies and timed or continuous
measurement was taken.
aspects directly related to interaction with the product, such
as the design of interactive elements, action possibilities, measurements in which different measurement
narrative structure [4], intended use and function [6]. qualities (as mentioned above) are combined to create a
Compositional aspects result in emotions (inner circle), picture of the overall UX or the dynamics therein (e.g.
such as (mis)understanding of a product’s workings, changes over time, at a specific moment in time, as a
(un)predictability of a product’s behaviour or outcomes,
• Enable researchers to perform situated measurements
mean over time).
and feelings of (not) making progress. Aspects related to
meaning are those through which a designer tries to realize to approach or realize UX measurements that reflect
the users’ higher order goals, by making users attribute users’ experiences in their every day lives.
meaning (middle circle) to the product. Through sense-
making processes people are able to recognize metaphors, The above requirements were used to develop four different
assign personality or other expressive characteristics, and
• Logging tools: automatic capture of user’s behaviour or
kinds of TUMCAT measurement tools [1]:
assess the personal or symbolic significance of a product
[6]. Attributed meaning can result in emotions (inner circle)
product usage (e.g., mouse-clicks, keystrokes,
such as anger, joy, satisfaction, fulfilment, fun, bliss,
• Sensing tools: automatic capture of a user’s context
application feature usage).
closeness to one’s own identity or image, inspiration and
regret [3, 5].
(e.g. physical, social and/or virtual such as physical
According to Wright & McCarthy [4], experiences come to location, buddy lists and buddies online, active

• Experience sampling2 tools: automatically generated

life via a process of sense-making. Sense-making refers to application window, system/application status).
processes such as: Anticipation, expectations about our
experience with a product; Connecting, our first experience self report requests sent once or multiple times to the
(sense of speed, thrill, openness, etc.) with the product and user, triggered by user activity (logged data), specific
its interaction without giving meaning to it; Interpreting, context aspects (sensed data) or a predefined time

• User generated content tools: provided to the user to

interpretation of our interaction ((non-) instrumental, (non-) schedule.
physical [6]) with the product, relating it to our goals,
desires, hopes, fears and previous experiences, leading to give feedback about any desired topic when a user feels
experiences such as anxiety, unease, desire, willingness to like it (e.g. feedback button linked to a feedback form,
continue); Reflecting, making judgements about our discussion forum related to the product).
experiences, evaluating them and comparing them with
other experiences, resulting in satisfaction, excitement or
boredom, a sense of achievement, etc; Appropriating, trying 2
Experience sampling is a set of empirical methods that are
to identify ourselves with the experience or changing our designed to repeatedly request people to document and
sense of self as a consequence of the experience; report their thoughts, feelings, and actions outside the
Recounting, reliving experiences by recounting them to laboratory and within the context of everyday life.
In: Proc. of the Intern. Workshop on Meaningful Measures: Valid Useful User Experience Measurement (VUUM),.Reykjavik,
Iceland, 18 June 2008, E.L-C. Law, N. Bevan, G. Christou, M. Springett, and M. Lárusdóttir (pp. 72-78)
4 CASE STUDY: TRIBLER
Peer-to-peer (P2P) networks are networked computer
devices (nodes, terminals) that permit other computer
devices to utilize locally available storage space,
communication bandwidth, processing capacity, and
sometimes even hardware components. P2P technology has
brought clear advantages over client-server architectures
[8]. Yet, the success of any P2P system fully depends on
the level of cooperation among users. Technical
enforcement of this cooperation is limited. Therefore,
within the Tribler project an alternative approach is chosen:
making use of knowledge from (social) psychology on
altruistic behaviour for developing cooperation inducing
features [8].
Figure 2. Tribler 4.0 main screen.
4.1 What is Tribler?
Responses to these questions were automatically sent back
Tribler is a peer-to-peer television (P2P-TV) system for
to the server. In addition, test participants had a possibility
downloading, video-on-demand and live streaming of
to provide user generated content to the research team by
television content [2]. The system not only gives users
clicking a feedback icon in the system tray of their
access to all discovered content and other users in the
computer and then entering text in a text window that
network, but also provides the means to browse
would pop up. By clicking the submit button this feedback
personalized content with a distributed recommendation
was sent to the server and was merged with the logged data
engine and an advanced social network that each user
file. Test participants were gathered through personal
creates implicitly and explicitly. An advantage of having
contacts, as well as through flyers and posters at institutes
trustworthy friends in Tribler is that they can speed up the
of higher education as well as technical institutes of applied
downloading process by donating their own idle bandwidth.
science. They were given a € 25 reward after the five weeks
Figure 2 shows the main screen of Tribler version 4.0.
had passed. In the fourth week of the study they were asked
to perform some specified tasks and to fill in a
4.2 Measuring experiences with using Tribler
questionnaire. Tasks and questionnaire were similar to
Measurements were taken by conducting a longitudinal
those of the laboratory study
field study using the TUMCAT measurement tools, as well
as by conducting a laboratory study in which test
4.2.2 Laboratory study
participants were asked to perform some tasks with Tribler
In the laboratory study ten test participants were asked to
and by asking international experts on usability and user
download the Tribler software, explore its use and then
experience from the COST294 network to provide their
perform some tasks with it, in the presence of a facilitator.
expectations on the users’ experiences with Tribler.
Sessions lasted about one hour to one hour and a half.
Example tasks were: accepting an invitation to become
4.2.1 Longitudinal field study
friends with someone; searching for a specific movie;
In the field trial 39 users were asked to download Tribler
asking a Tribler friend to donate bandwidth to speed up
and use it at home for five weeks in any way they wanted.
downloading. Tasks were designed to make users use basic
Together with their download of the Tribler software they
downloading functionality, as well as features that
also received a customized version of the uLog3 software
distinguish Tribler from other competing file sharing
that logged all activities users performed with Tribler. An
software (e.g., recommendations, donating bandwidth to
other tailor-made software package sent the loggings to a
friends). Users were asked to think aloud while the
central server that the researchers used for data gathering.
facilitator was sitting next to them. The facilitator took an
This software also sensed application and user related data
active listener approach [9] and asked questions at
that Tribler stores on the user’s computer (e.g., a list of the
appropriate times to probe the user’s understanding of the
user’s friends on Tribler and the library of downloaded
software’s functionality, e.g., Have you noticed the
files). To be able to gather subjective data from the test
checkmark / exclamation mark in the top right corner?
participants, the server also sent out Experience Sampling
What does it mean to you? Retrospective interviews were
questions to users which opened in a browser window.
conducted to further discuss some of the events or problems
These questions were automatically triggered based on
that came up during sessions and users were asked to fill in
specified logged activities or combinations of activities.
the Attrakdiff4 questionnaire on paper.

3
For more information on the uLog software see:
4
http://www.noldus.com/site/doc200603005 For more information see: http://www.attrakdiff.de/
4.2.3 Expert review session: Would you consider starting to use Tribler at
In the expert review five experts were provided with the home? How frequently? What would you use it for? Under
Tribler software to review it. They were asked various what circumstances? In the expert review, the experts were
questions assessing their opinions on the software. They asked similar questions: 1) after a quick exploration of
were also asked to estimate what problems users would Tribler: Do you think that the target group may indeed
have with the software, and to fill in the Attrakdiff consider downloading, or actually download the software
questionnaire imagining what the users would answer. once they are aware of its existence? What do you think
Communication with the experts was solely through email. users would want to use Tribler for in particular? 2) in
relation to Tribler’s friends and recommendations facilities:
4.3 Measurements in the different studies Do you think that (in the long term) Tribler users will
The various types of measurements taken are discussed actively engage in using this facility?
using the UX framework explained in section 2, focusing
on compositional, meaning and aesthetics UX aspects. It is 4.3.2 Attractiveness
assumed here that in case of software for voluntary use its One of the ways of measuring attractiveness was by having
attractiveness largely determines whether or not and to what the test participants fill in the Attrakdiff questionnaire. This
extent the software will be used. The attractiveness of using questionnaire is based on a theoretical model in which
the software is closely related to the emotional response to pragmatic and hedonic aspects of experiences are thought to
(the exposure to and interaction with) the software. In affect the attractiveness of a product. A number of
addition, factors like the user’s context, a user’s pre- questions assess the product’s attractiveness for the user. In
dispositions and constraints play an important role (e.g., the field test, participants who had actually used Tribler
familiarity and availability of other software with similar were asked to fill in the questionnaire in the fourth week of
functionality, previous knowledge of P2P file sharing their use. In the laboratory study participants filled in the
software, technicalities and compatibility of the user’s questionnaire after their task performance. In the expert
system). view, the experts were asked to fill in what they thought
The sense-making process of the UX framework (depicted users would fill in.
in the outer circle) implies that there are temporal issues In addition to this questionnaire, participants were also
involved. One has to take into account the user’s experience asked about their appreciation of specified Tribler features
in relation to the product prior to or at the very start of the on a 5-point scale. In the field study, these questions were
actual interaction (e.g., anticipation and connecting), as asked as experience sample questions in response to the use
well as during and after the interaction (i.e., the other of the specified function (e.g., after the 1st and every 5th
elements of the sense-making process). time of using the function). In the laboratory study all
Data was gathered on usage and on the attractiveness of the participants were asked to answer the questions after having
software, in addition to data that related to the different UX performed a task related to that function. In the expert view,
aspects. Temporal issues were addressed in different ways experts were instructed to answer the questions imagining
in the three types of studies. In the field trial data gathering how they thought users would appreciate the specified
on usage was automatically done over the five week study functionality.
period. In case of the laboratory study, users were asked Finally, insight into the users’ emotions in reaction to the
UX and usage related questions after their initial product were received through spontaneous (written)
experiences with Tribler, as well as during the whole feedback by the participants (field test) as well as through
session and afterwards, imagining future usage. In the the retrospective interviews and observed (verbal and no-
expert review experts were asked to provide their verbal) reactions during task performance (laboratory
expectations on the UX after their first confrontation with study).
the software, as well as after further inspection and trial.
4.3.3 Compositional aspects, meaning and aesthetics
4.3.1 Usage Compositional aspects relate to the pragmatic aspects of
In the field study, TUMCAT’s automated logging facilities interactions, including usability problems, effectiveness etc.
made it possible to monitor actual usage of Tribler at the In all three studies measurements included those questions
level of UI events (e.g., shifts in input focus, key strokes) of the Attrakdiff questionnaire that related to the software’s
and the abstract interaction level (e.g., providing values in pragmatic quality. In the field study, the logged and sensed
input fields) [10], providing insight into how usage data in combination with spontaneous user feedback shed a
developed over the five week time period. In the laboratory light on pragmatic issues. In the laboratory study such data
study test participants were asked to imagine whether and were gathered by observing task performance and through
how they would use Tribler at home in three different ways: retrospective interviews. In the expert review, the experts
1) after a short exploration of the software: what kind of were given the tasks used in the laboratory study as
things do you think you would use Tribler for at home; 2) suggestions to structure their search for usability problems
after each task: would this be something you can picture in the software and were also asked to provide some
yourself doing at home? (scale 1-5) and 3) at the end of the reasons on why they thought a problem would occur.
In: Proc. of the Intern. Workshop on Meaningful Measures: Valid Useful User Experience Measurement (VUUM),.Reykjavik,
Iceland, 18 June 2008, E.L-C. Law, N. Bevan, G. Christou, M. Springett, and M. Lárusdóttir (pp. 72-78)

7,0
data from the laboratory study, as well as from the expert
Lab t est review provided a rich view on problems and their possible
Expert Review
6,0 Field t est
causes. This was the kind of data that designers in the
Tribler development team could most readily use in their
5,0
attempts to redesign Tribler. Aspects of meaning were
found mainly in the laboratory study through spontaneous
4,0
think aloud utterances, as well as in retrospective
interviews. This related to issues like terminology being too
3,0
dull for them, not wanting to use social software, not
valuing recommendations of files based on popularity,
2,0
issues in relation to (appreciating or not appreciating)
illegal downloads and (laughing about or feeling offended
1,0 by) adult content. In the expert review two experts
PQ HQI HQS ATT
commented on issues of meaning, only in relation to not
appreciating illegal content and feeling offended by adult
Figure 3. Attrakdiff results (PQ: pragmatic quality, HQI: content. In the field study only two users commented on
hedonic quality (identification), HQS: hedonic quality issues of meaning. They did so in the same way as the
(stimulation), ATT: attractiveness. 1 indicates low score, 4 experts in the expert review.
indicates neutral. Lab test (n=11), Expert view (n=5), Field test
(n=6)). As to the aesthetic aspects, three experts mentioned issues
of graphical design and layout in their comments. Generally
In addition, the experts were explicitly asked about whether they indicated they liked the graphical design, although also
they thought users would understand the logic behind once ‘bad layout’ was mentioned, as well as being disturbed
specified functions. Aspects of meaning were measured by about the software not showing thumbnails in the files
the questions in the Attrakdiff questionnaire that related to view. In the field test only two participants commented on
the software’s hedonic qualities (all three studies). It was aesthetic aspects, mentioning they disliked the library and
also expected that in all three studies spontaneous feedback files view. In the laboratory study a rich mix of comments
from users and explanations on expected future usage was given (by 7 participants) on aesthetic issues. Opinions
would provide some insights on these aspects. Aesthetic here were more mixed in the sense of valuing the design or
aspects were not explicitly included in any of the not, but also of the level of design detail they commented
measurements of the three studies. However, spontaneous on, ranging from comments on specific icons to an opinion
remarks or feedback could provide some data on such on the general looks of the software. Many of the comments
aspects. were spontaneous exclamations when confronted with a
new screen.
5 PRELIMINARY RESULTS AND ANALYSIS
A preliminary analysis of the data was conducted, which From the Attrakdiff questionnaire we found that pragmatic
allows drawing some tentative conclusions on how the aspects in the field trial and the laboratory test were more or
findings from the three studies relate to each other in terms less the same and scored more negatively than in the expert
of UX framework elements. review. Hedonic identification with, and stimulation by
Tribler scored lowest in the field test, followed by the
From the field study it became clear that 15 of the 39 test laboratory test and the experts’ opinions. Attractiveness
participants had started using Tribler during the five week scored similar over the different studies, see figure 3 for an
period; these were given the questionnaire. Only 6 of them overview.
filled it in, being active users in the sense that on occasional
days they spent some hours on using the software. From 6 DISCUSSION AND CONCLUSIONS
questions asked to the 15 ‘real’ users and through informal Analysis of these studies is still preliminary and ongoing. In
communication it became clear that reasons for not using the following we describe our first findings. We can only
Tribler had to do with software packages crashing, the gain insight in the actual usage of Tribler and its’ functions
combination of software making computers too slow, as by using tools such as logging and sensing over a longer
well as them being used to competing software. Statements period of time. Laboratory tests and expert reviews give
participants in the laboratory study made, predicted that weak predictions about usage which do not agree with the
utility and usability problems (compositional aspects) results from logging and sensing. Logging and sensing do
would prevent some people from using the software. In the not provide any explanation for the actual usage. We found
expert review similar views were expressed. A superficial lower scores in the field test for both meaning and hedonics
analysis of the logging and sensing data gathered in the (from the Attraktdiff questionnaire) than from the
field study could not provide a clear view on compositional laboratory study or the expert reviews. Actual usage makes
aspects like usability problems; it was too difficult to trace people aware of the match between product and their higher
back what users were trying to do from mere log data. The order goals (or meaning). This may account for this result.
Laboratory tests and expert reviews give a detailed and rich ACKNOWLEDGEMENT
insight in compositional, pragmatic issues such as the We would like to especially thank the following people for
usability. Logging and sensing do not provide this detailed their contributions to these studies: Gilbert Cockton, Effie
insight. Logging tools used, monitored user activity on the Law, Jens Gerken, Hans-Christian Jetter and Alan
levels of UI events (e.g., shifts in input focus, key events) Woolrych from the COST294 network. Ashish Krishna and
and the abstract interaction level (e.g., providing values in Anneris Tiete for their contribution in executing and
input fields). To generate information about usability analysing the study and the preliminary results. The test
issues, the data needs to be transformed to higher levels of participants for their active contribution and feedback. Leon
abstraction such as domain or task related levels (e.g., Roos van Raadshoven, Paul Brandt, Jan Sipke van der Veen
providing address information) or to goal and problems and Armin van der Togt for their technical contributions.
related levels (e.g., placing an order) and compared to
predefined or automatically identified sequences of user REFERENCES
activity within these levels [10]. 1. Vermeeren, A.P.O.S. and J. Kort, Developing a testbed
for automated user experience measurement of context
Attractiveness scores were slightly negative on the aware mobile applications, in User eXperience, Towards
Attrakdiff questionnaire in all three studies. Though the a unified view, E. Law, E.T. Hvannberg, and M.
score was slightly negative we doubt if this is a valuable Hassenzahl, Editors. 2006, COST294-MAUSE: Olso. p.
predictor for the (non-)usage as measured during the field 161.
test (it seems too bold that a slightly negative score can
result in such overwhelming non-usage). The influence of 2. Fokker, J.E., A.P.O.S. Vermeeren, and H. de Ridder,
attractiveness on usage might be dependent on people’s Remote User Experience Testing of Peer-to-Peer
motivation to use a product (externally motivated or Television Systems: a Pilot Study of Tribler, in
internally motivated product interaction). Aesthetic aspects EuroITV'07, A. Lugmayr and P. Golebiowsky, Editors.
are difficult to measure for interactive products. We found 2007, TICSP, Tampere: Amsterdam, The Netherlands.
attractiveness as well as aesthetics are best measured p. 196-200.
through direct interaction with participants. For experts it’s 3. Kort, J., A.P.O.S. Vermeeren, and J.E. Fokker,
difficult to formulate an opinion about attractiveness and Conceptualizing and Measuring UX, in Towards a UX
aesthetics from a user’s viewpoint. Manifesto, COST294-MAUSE affiliated workshop, E.
Law, et al., Editors. 2007, COST294-MAUSE:
7 FUTURE WORK Lancaster. p. 83.
Based on the conclusions above, we identified several 4. McCarthy, J. and P. Wright, Technology as Experience.
issues for future work on improving UX measurement in 2007: The MIT Press. 224.
long-term field studies:
5. Pals, N., et al., Three approaches to take the user
1. Tools or methods to raise the level of logging data perspective into account during new product design.
from the UI events and the abstract interaction level to International Journal of Innovation Management, 2008.
higher levels such as domain and task or even goal and In press.
problem levels and means to analyze the data on these
higher levels (detecting sequences and interpreting 6. Desmet, P. and P. Hekkert, Framework of Product
these sequences); Experience. International Journal of Design, 2007. 1(1):
p. 10.
2. Approaches for automated gathering of data that 7. Hekkert, P., Design Aesthetics: Principles of Pleasure in
provide (a) insight into reasons of (non-) usage at the Design, Delft University of Technology, Department of
level of products or product features, (b) insight into Industrial Design: Delft. p. 14.
why or how the product succeeds (or not) in making
the user attribute meaning to it; (c) rich and detailed 8. Pouwelse, J.A., et al., Tribler: A social-based peer-to-
data on usability issues, especially those that relate to peer system. Concurrency and computation: Practice
longer term usage and are highly affected by the and experience, 2008. 20(2): p. 127-138.
personal situation of the user. Especially for topic 2a 9. Boren, T.M. and J. Ramey, Thinking aloud: reconciling
more detailed theoretical knowledge in the area of UX theory and practice. IEEE Transactions on Professional
would help to relate the various aspects in the Communication, 2000. 43(3): p. 261-277.
framework to each other and to a product’s
10. Hilbert, D.M. and Redmiles D.F, Extracting usability
attractiveness and (non-) usage in real-life.
information from user interface events. ACM
3. A practical approach for assessing the aesthetic aspects Computing Surveys (CSUR). 2000. 32(4): p. 384 – 421.
of a product.