User Testing Report of Kayak website

Created by Ying Li for Coursera course_ Evaluating Designs with Users

Updated: July 7, 2022

1
 Executive Summary

This research aims at evaluating users’ experience when using Kayak to plan their trips,
including booking flight tickets, making hotel reservations, and setting a fare alert to keep track
of price changes. The method of this research is to use usability testing to ask participants to
accomplish a set of tasks (with Think Aloud). The researcher started with several pretest
interviews to collect participants’ demographic information, travel experience, and booking
preferences. The 2 usability tests in this research lasted 35-45 minutes and asked participants to
complete 6 different tasks. Participants were also asked to speak out about what they were
thinking and doing when completing the tasks. After the test, the researcher used the SUS
questionnaire to collect general opinions and attitudes of users and conducted follow-up semi-
structured interviews to explore what happened during the task session and their
opinions/feelings about their user experience.
Through analyzing the recorded task session, the narratives from Think Aloud, and the post-
task questionnaire and interviews, the researcher identifies several key problems in user
experience, including the price/time adjust bar, loading speed, and delayed response when
making changes to the search. The researcher recommends adding other options under the time
adjust bar to let users directly put in numbers; accelerating loading speed by eliminating the
search and comparison of prices from some unnecessary websites; providing users feedback and
updated results when users make changes in the middle of a search.

2
 Introduction

Kayak is an American online travel agency and metasearch engine. The search covers a variety
of fields, including flight tickets, hotel reservations, car rentals, local activities, and trains and
buses. The essential and even unique function of Kayak is to search for the lowest price across
different websites. This research aims at answering two key questions: (1) What are some
difficulties/frustrations users confront when searching and finding the results that meet their
needs? (2) What are some strengths that the system needs to maintain and further improve to
ensure users will recognize and choose Kayak for future needs? These questions are very critical
because they can uncover potential problems and flaws of the system that might have led to the
loss of customers and reveal the positive values of the system from real users’ perspectives to
continuously attract more users.
The usability tests conducted in this research aim at revealing the process of searching,
finding, and interacting with different functions of Kayak from real users. The post-test
questionnaires collect data on how users evaluate the website and their preferences on reusing
the website and recommending the website to their friends. The post-test interviews further
illustrate essential difficulties and problems users encounter during the test and explore why
those troubles happened. The data collected will help researchers to answer the key questions
stated above and stimulate possible plans for improvement and redesign.

3
 Methods
The target population of this research is users who have experience using an online website to
search and plan their travels in the past year but are also first-time users of Kayak. The
researcher recruited participants through convenience sampling by asking friends and family
members. Two participants are recruited for this research. The first participant is a young
professional (Gender: Male, Age: 35) that has experience using different search engines in the
past year, including Expedia and TripAdvisor. He is familiar with mainstream travel search and
book engines online. He never used Kayak to search for travel information but has heard about
this website before. The second participant is a housewife (Gender: Female, Age: 32) who has
been taking care of her daughter for the past 3 years. She used other travel search and booking
websites to book international flight tickets for her mother 4 times and domestic flight tickets,
hotel reservations, and car rentals 3 times in the past two years. She never heard about the
website Kayak, so this was her first trying the website.
The major methods of data collection included semi-structured interviews, usability tests,
Think Aloud, and questionnaires. The research started with short warm-up interviews with
participants to collect their demographic information, travel experience, and booking
preferences. The 2 usability tests in this research lasted 35-45 minutes and asked participants to
complete 6 different tasks. The tasks were arranged from easy to difficult that covered different
functions of the system, including searching, finding satisfying results, adjusting restrictions, and
keeping track of selected information. Participants were also asked to speak out about what they
were thinking, feeling, and confusing while completing the tasks. After the test, the researcher
used the System Usability Scale (SUS) questionnaire to collect data on how generally users feel
about the system. The researcher then conducted follow-up semi-structured interviews to explore
what happened during the test and why those difficulties and troubles happed.
The data analysis first takes advantage of the “Affinity Wall” method. The researcher
reviewed the recorded test sessions, including the navigation process, the action flow, and the
Think Aloud narratives, and wrote down critical observations, quotes, facts, opinions, and
questions on different sticky notes to create an “Affinity Wall.” Then the research summarized
and reorganized the notes into major categories. Combining knowledge and principles of
“Heuristic Evaluation,” the research figured out the key problems of the system related to users'
troubling experiences. The narratives from follow-up semi-structured interviews further provide
insights into how specific problems emerged during the tests related to a specific design of the
website and why some troubles happened. Depending on those findings, the researcher provides
some recommendations to address those problems and improve the user experience of the
system.

4
 Findings and Recommendations
Summary Results
Both participants successfully completed 6 tasks in a reasonable time. The scores of the post-test
questionnaire (SUS) are 90 and 75, which are both the average score of 68. According to
research, scores above 68 indicate need for minor improvements to the design. The usability tests
also reveal that both users had smooth processes of completing the tasks most of the time. The
research still identifies some problems that can be addressed in future redesign and upgrade of
the system, including increase the loading speed, change the format of specific functions
bars/tabs, and give users more freedom of taking actions in the middle of a search.

Key Findings and Recommendations

Key Finding #1: The adjust bar to set the take-off time range at the departure and destination is
difficult to use.
[Evidence]: During the test, both users spent much time to adjust this time bar to meet the
requirements listed in the tasks. Researcher observed that users got a little bit frustrated when
failed to adjust to the right time range the first one or two times of trying. The first user even
said, “OK. It seems that I could not accurately get it to 9 am. I will just leave it as 9:45 am. Why
can we just input the time?”
Recommendation: It is nice to have the adjust bar to change time range. A good feature of
this is that when you put the mouse on the circle button to change time, a chart will automatically
show up and tell users the level of business at different times. I would suggest to keep this bar,
but at the same time, add two blanks for users to directly enter the time they want to set.

5
Key Finding #2: The loading speed when doing the search is a little bit slow. While waiting for
the results showing up, users were facing a completely blank page until the very end of the
search.
[Evidence]: When users put in criteria and start a search, it usually takes 5-10 seconds to
have the results showing up. I observed that users got a little bored while waiting for the results
to show up. Participant 1 said, “It is a little bit slow.” Participant 2 said, “That’s too slow. There
is still no result pop up.”
Recommendation: I have two recommendations for this. During my follow-up interviews,
one participant said, “I understand that it takes some time to compare prices from different
websites. But it is a little bit slow. Maybe just leave out some small websites, or the ones have
overlapping resources, so it can save some time.” The user’s suggestion provides a possible
direction to work on. Another suggestion is in the middle of search, it would be great to show
some results that are already being found, so users can start to look at some information instead
of waiting and staring at a blank page.

6
Key Finding #3: In the middle of a search, when users realize that they need to change the
searching criteria, they usually immediately put in the new information. However, the new
search will not automatically replace the older one. Users must re-click the search button to start
the new search. If not, the results will not be updated. However, users expect that the results can
be automatically updated after they typed in new information and searching criteria.
[Evidence:] Participant 01 realized that they made a mistake by putting the wrong dates that
did not match the dates required in the task. The user tried to make a change when the first round
of searching was processing. The user directly changed the dates without clicking the search
button again. The results from the first round of search have popped out. However, the user
thought the results were updated and started to take a deeper look at the results. After a while, the
user realized that the results were not correct and spent extra time to check search criteria and
started a new search. In the follow-up interview, the user brought this issue up again and
described this mistake he made. The user expressed feelings of frustration when realized his
efforts and time were wasted.
Recommendations: Provide efficient feedback to users when they had new interactions with
the system, such as changing the searching criteria. It would be great if the search can be
automatically updated and provide the correct results to users immediately.

7
 Limitations
One limitation of this research is that number of participants is limited. Only two participants
were recruited. This small sample size might not be sufficient and convincing to represent larger
group of users. Also, both recruiters are young people who share many similarities in using
internet and travel search and booking websites. The research might miss some important
insights from other groups of people, such as more aged people.

8
 Conclusion
This research illustrates that Kayak is a good website for users. No significant flaws and
problems are identified, but other smaller issues are uncovered and need to be fixed or improved
to create better user experience. The key findings of this research reflect on difficulties in using
the time adjust bar, slow loading time of search, and delayed response to changes. Future
research should expand sampling size and variety to further confirm the significances of existing
issues and reveal some other potential problems.

References
Nielsen, J. (1994) Heuristic Evaluation. In J. Nielsen. & R. L. Mack (Eds.) Usability
Inspection Methods. New York, NY: John Wiley & Sons.

9
 Appendices
Test Script
Pretest Checklist
● Clear cookies
● Login with newly registered email: [email protected]; password: qq1234
● Remove login saved memory
● Double check success criteria
● Fill in the blanks in this Test Script, User Testing Plan, and Consent Form with the name of the website
● Print task instructions (one per page)
● Print post-test questionnaire, follow-up semi-structured interview protocol, logging sheet, test script
and consent form
● test recording through Zoom
● Start screen sharing and meeting recording

Posttest Checklist
● Stop recording, save audio and video to backup drive
● File informed consent form
● File logging sheet

Introduction to demo
Thanks for coming in today! We’re constantly trying to improve our product and getting your frank
feedback is a really important part of that. The goal for today’s session is test the website: Kayak, a
travelling search engine and information platform. I’m here to learn from you so I’ll ask a lot of
questions, but I’m not testing you. There are no right or wrong answers.
I’ll start this session by asking some background questions. Then I’ll show you some things we’re
working on, and ask you to do some tasks. As you work on the tasks, please think aloud.
This means that you should try to give a running commentary on what you're doing as you work
through the tasks. Tell me what you're trying to do and how you think you can do it. If you get
confused or don't understand something, please tell me. If you see things you like, tell me that
too. I want to emphasize that; you won’t hurt my feelings by telling me what you think. In fact, any
feedback will be helpful.
If you do get stuck, I’m going to try not to answer your questions or tell you what to do. I’m
just trying to see what you would do if you were using it on your own. But don’t worry-- I’ll help
you if you get completely stuck.
Do you have any questions before we begin?

Consent Form
Present Consent form, summarize it, and obtain signature

Pre-test Warm-up interview

1. Have you used Kayak before? Have you heard about this website before?
2. Tell me about the last trip you planned.
a. What do you usually use to plan your trip?
b. What is your primary purpose for travelling?
c. What is your primary concern?
d. What is your budget?

10
3. What information is the most important when you are planning your trip?
4. How often do you travel?
5. Some demographic information

Task Instructions
Print tasks and present them, one at a time. Read each task aloud and give the printed sheet to
the participant.

Post-test Questionnaire (SUS)

Hand the printed questionnaire to the participant. Explain how the questionnaire works and give
participant some privacy to complete the questions.

Follow-up Semi-structured Interview

1. Review the questions together with the participant and ask why they gave a lower score for some
questions.
2. Review parts of the test where the user struggled: What difficulties did you
have on ____? I noticed you struggled with____, can you tell me what happened? You
paused here, tell me more about that.
3. Preferences: What did you think of the site? What did you like/dislike?Which
parts of this page are most/least important to you?
4. Changes: If you had 3 wishes to make this better for you, what would they be?
Why?
5. Understanding: How would you describe this to a friend?
6. Use Cases: Under what circumstances would you use this? Why?

Conclusion
This has been incredibly helpful. Today, you mentioned…[Researcher: Try to briefly summarize
some key parts of the discussion or issues.] Your input is really valuable for me and the team as
we think about the next steps for these ideas. We really appreciate your taking the time to come
in, and answering all of my questions. Thanks SO much!
[Moderator: Give participant incentive gift, if appropriate.]

Your consent form (unsigned)

I agree to participate in the study of a flight booking website being conducted as part of the
Coursera course: Evaluating Designs with Users.
I consent to the recording of this test. This recording will be used for research and product
improvements only.
I understand that participation in this usability study is voluntary and I agree to immediately raise
any concerns or areas of discomfort during the session with the study administrator.
Please sign below to indicate that you have read and you understand the information on this
form and that any questions you might have about the session have been answered.
Date:_________
Please print your name:
____________________________________________________
Please sign your name:
____________________________________________________ Thank you!
We appreciate your participation.

11
Any screening, pre-test, post-test questionnaires that were used
SUS (System Usability Scale)
Questions Strongly Somewhat Neither agree Somewhat Strongly
disagree disagree or disagree agree agree
(1 point) (2 points) (3 points) (4 points) (5 points)
I think that I would like to use this
[project] frequently.

I found the [project] unnecessarily

complex.
I thought the [project] was easy to use.

I think that I would need the support of a

technical person to be able to use this
[project].

I found the various functions in this

[project] were well integrated.

I thought there was too much

inconsistency in this [project].
I imagine that most people would learn to
use this [project] very quickly.

I found the [project] very cumbersome to

use.
I felt very confident using the [project].
I needed to learn a lot of things before I
could get going with this [project].

Filled-out logging forms, or a more readable transcription of the data from them

Test Name: Usability test of Kayak

Participant ID: 01
Task Number: 6
Logger: Ying Li
Date: 07/02/2022
Task Success: 6 tasks completed

Clock Time Event Code Notes

2:00 pm Task start Start task one. User smoothly completed the task.
2:05pm Task 2 The user easily finished the task, but used more time.
2:11 pm Task 3 The user spent much longer time reading the task to make
sure he understood the task right.
2:24 pm Task 4 The user spent much time adjusting the time bar. He failed
several times to get the right time. He gave up in the end.
2:38 pm Task 5 The hotel task if quite easy.

12
 2:43 pm Task 6 User spent some time understanding what a fare alert is. He
tried to find the function on the front page. He noticed the
information when he finished the search.
2:50 pm Task end

Test Name: Usability test of Kayak

Participant ID: 02
Task Number: 6
Logger: Ying Li
Date: 07/07/2022
Task Success: 6 tasks completed

Clock Time Event Code Notes

8:06 pm Task start Start task 1 easily
8:10 pm Task 2 Start the 2nd task
8:15 pm Task 3 Realized a mistake in the searching criteria. The user tried to
make a change by putting the new information, but the
searching results did not update accordingly. The user was a
little bit frustrated.

8:21 pm Task 4 Difficulties in using the time adjust bar.

8:32 pm Task 5 Complete smoothly
8:40 pm Task 6 Spending a lot of time figuring out where to find the fare alert
function.
8:47 pm Task end

Raw questionnaire responses

Participant 01
Questions Strongly Somewhat Neither agree Somewhat Strongly
disagree disagree or disagree agree agree
(1 point) (2 points) (3 points) (4 points) (5 points)
I think that I would like to use this
[project] frequently. 

I found the [project] unnecessarily

complex. 

I thought the [project] was easy to

use. 

I think that I would need the

support of a technical person to be 
able to use this [project].

I found the various functions in

this [project] were well integrated. 

I thought there was too much

inconsistency in this [project]. 

13
I imagine that most people would
learn to use this [project] very 
quickly.
I found the [project] very
cumbersome to use. 

I felt very confident using the


[project].
I needed to learn a lot of things
before I could get going with this 
[project].

Participant 02
Questions Strongly Somewhat Neither agree Somewhat Strongly
disagree disagree (2 or disagree agree agree
(1 point) points) (3 points) (4 points) (5 points)
I think that I would like to use
this [project] frequently. 

I found the [project]

unnecessarily complex. 

I thought the [project] was easy

to use. 

I think that I would need the

support of a technical person to 
be able to use this [project].
I found the various functions in
this [project] were well 
integrated.
I thought there was too much
inconsistency in this [project]. 

I imagine that most people

would learn to use this [project] 
very quickly.

I found the [project] very

cumbersome to use. 

I felt very confident using the


[project].
I needed to learn a lot of things
before I could get going with this 
[project].

14
A complete list of usability issues found, with severity ratings

Issues identified Severity Ratings

Difficulties in using the time adjust bar 3
Slow loading time of search 3
Delayed response when making a change in the middle of a search 3
Difficulties in find the “fare alert” 2
Too many banners that are a little bit distracting 2
Users must click each icon to find out the meanings on the left bar on the 2
front page

15