SVKM’s NMIMS

UI UX

4th Year
Aim: To implement quantitative and qualitative research design techniques for
<<your project >>

Theory:

Trained UX researchers have a variety of tools at their disposal to help take the guesswork
out of UX design. Their methods generally fall into one of two categories:

 Qualitative research, which helps them understand user attitudes and behaviors; and

 Quantitative research, which helps them understand the statistical significance behind

experiences

Quant methods allow you to:

 Put a number on the usability of your product. Numbers are sometimes more
persuasive than findings and videos from qual testing (particularly when you’re trying
to convince folks like executives).

 Compare different designs (for example, your new version vs. your old version, or
your product vs. your competitor’s product), and determine whether the differences
you observe are statistically significant, and not due to random chance.

 Improve UX trade-off decisions. For example, if a proposed design improvement is

expected to be expensive to implement, is it worth doing? If you have an estimate of
how much the change will improve the usability, a quant method may help you decide
whether the redesign is worth it.

 Tie UX improvements back to organizational goals and key performance indicators

(thus demonstrating your return on investment and justifying your UX team’s
existence).
One way to think of qualitative research is that it answers the “why?”—why are users
struggling to complete tasks? Why does the number of users fall off after a certain time? Why
is the conversion rate so low?

In comparison, qualitative research helps in answering the question “How much and how
many?”—it’s based on math, web analytics, and big data, and is primarily focused on
measuring the size and scale of a problem.

What Is Qualitative UX Research?

Qualitative research, also referred to as “qual,” is the process of collecting and analyzing non-
numerical data to understand user attitudes and motivations, and developing an in-depth
understanding of a problem. Using qualitative research methods, UX researchers try to
answer questions such as, “why?” and “how?”

Qualitative research generates insight into how users experience a product or service, and
researchers will often use tools such as open-ended customer surveys, focus groups,
interviews, and observational field studies.

Unlike quantitative UX research, qualitative research doesn’t require as many respondents

because, instead of focusing on statistical significance, it seeks to understand the experiences
of users and formulate theories or hypotheses that can then be tested.

What Is Quantitative UX Research?

Quantitative research, also referred to as “quant,” is the process of collecting and analyzing
numerical data to identify patterns and averages, measuring data points, and producing results
that, as the name suggests, are quantifiable. Using quantitative data, UX researchers answer
questions such as “how many?”, “how much?”, and “how often?”

Quantitative research provides hard data, and researchers working with quantitative methods
often use resources like Google Analytics to track information such as user clicks, completion
rates, and conversion rates, time spent on a platform, and the results from A/B testing, etc.

Quantitative methods
1. Technical analysis. One of the most common types of quantitative user research is the
process of diving into user analytics to see whether any problems show up in the data. For
example, Google Analytics collects troves of data ranging from when people visit a website
to the time spent on a site, pages visited, referral details, task times, browser type, scroll
depth, and errors, when users try to fill out a form. From this information, UX researchers can
quickly identify trends and pinpoint where a problem might exist.

2. Mouse tracking analysis. Mouse tracking follows a user’s movement on a website. With

enough research participants, this methodology can be used to generate a heat map that shows
parts of a website where there is lots of activity and parts that get ignored. This is useful in
identifying basic usability issues, such as whether users are mistaking certain graphics for
clickable buttons, or whether they are trying to enter text into non-existent fields. It can also
help researchers determine scroll depth—understanding how far most users scroll can help
them figure out where they need to place the most important information.

3. Funnel analysis. Similar to technical analysis, funnel analysis focuses on how many users
complete a workflow and drop-off percentages for each stage of the funnel. For example,
when applied to an e-commerce platform, the funnel may include steps such as visiting the
shopfront, browsing, adding items to a virtual shopping cart, filling out a check-out page, and
completing an order. Funnel analysis allows UX researchers to zero in on where users might
be falling off, which can lead to more specific analysis.

4. User testing. This method, which invites real users to offer feedback, can be used to collect
both qualitative and quantitative data. In the latter, a researcher might record a user as they
perform certain tasks, measuring how long it takes for them to complete a task and noting
where a user runs into problems. The quantitative data collected can be used for
benchmarking and determining whether a redesign is necessary, or whether recent changes
made are an improvement.

Qualitative methods

1. User interviews. One-on-one interviews or focus groups with users are an effective way of
gathering qualitative data that sheds light on user attitudes and behaviors. Where quantitative
methods might pinpoint that a problem exists in the user funnel, speaking directly with users
can explain the nature of that problem—are shoppers stopping short of checking out because
of a glitch? Does the shipping cost cause them to change their mind? Is there something about
the check-out page’s design that disrupts their shopping experience? Interviews give
researchers the opportunity to go deep into the user experience.

2. User surveys. Surveys can be used in both qualitative and quantitative user research
depending on the nature of the questions. Open-ended surveys are often used in qualitative
research because the questions tend to be broad and invite users to go into detail about their
experiences. In this sense, they are similar to user interviews and focus groups, with the
added benefit of researchers being able to send them to a large number of people.

3. User testing. Where quant user testing might focus on how long it takes users to complete
tasks, qualitative user testing asks users to narrate their experience as they navigate a website,
platform, or app, offering their thoughts about how difficult it is to complete a task and how
they perceive the website (e.g. Are they trying to click on a graphic because they think it’s a
URL? Are they abandoning their shopping cart because of load times? Do they keep
canceling a transaction because they are accidentally interacting with pop-ups?)

4. Diary studies. Diary studies track user behaviors over a period of time and allow

researchers to collect data even when they are not actively interviewing or recording a user.
Some of the upsides of diary studies are that they minimize the bias that comes with a
researcher looking over a participant’s shoulder, they allow the user to participate in the study
in their own time, and they provide data points collected from a longer period of time.
Possible use cases of diary studies include having participants track each time they use an app
to order take-out—what was their motivation for using Seamless or Grubhub instead of
calling the restaurant directly? Why did they decide to get takeout instead of dining in or
doing an in-restaurant pick-up? Participants might also track every time they order running
shoes from a retailer—was the purchase inspired by a seasonal change? Does it align with a
new year’s resolution? Is it intended as a gift? Understanding these kinds of user motivation
can help a business better serve its customers.

For Nominal Data — Frequency, Chi-square

Simple descriptive statistics can be used for nominal data. For example, you can count the
numbers for each category and make a frequency table. To compare nominal data, you can use
a statistical test called chi-square. Chi-square test is used to decide if two categorical data are
related or unrelated (it’s called “dependent” or “independent” in statistics). For example, if
you want to compare the preferences between different user groups, then you conduct chi-
square test to see if the difference between user groups are significant or not. If you use Excel,
you can use the function “CHITEST” to do the chi-square test easily.

For Ordinal Data — Frequency, Chi-square, Wilcoxon Rank Sum Test

Looking at frequencies is a common way to analyze ordinal data. For example, 20% of users
rated the design excellent, 40% of users rated good, and so on. The differences between each
order are meaningless, so you can’t calculate average ratings. For more advanced analysis,
there is a method called Wilcoxon Rank Sum Test which is used to compare ordinal data.
Wilcoxon test can be done using Excel, but it would be easier if you use a programming
language such as R.

For Interval data — All descriptive statistics, T-Test, ANOVA, Correlation

Interval data allow you to use wide range of descriptive statistics, such as average and
standard deviation. Also, you can use some inferential statistics to derive a general conclusion
that applies to larger population, not limited to your test participants. One of the most common
ways to analyze interval data is comparing the means(averages), using T-test or ANOVA. T-
test is used to compare two samples; ANOVA is used to compare three or more samples. If
you use Excel, you can simply use the function “TTEST” for T-test. It’s also possible to
do ANOVA with Excel, using Analysis ToolPak add-in.
Scatterplot with trend line

Another useful way to analyze interval data is looking at relationship between different
variables. The chart above is an example of scatterplot with trend line, showing the correlation
between two variables. You can calculate correlation coefficient to see how the two variables
are correlated. Excel has “CORREL” function to calculate correlation coefficient, as well as
chart functions to draw scatterplot with trend line, including r-squared value that shows how
strongly the values are correlated (r-squared is simply the square of the correlation
coefficient).

For Ratio Data — All descriptive statistics (including geometric means), T-Test, ANOVA,
Correlation

There are not much difference between interval data and ratio data, and all the statistics that
are used for interval data can also be used for ratio data. One difference is that you can
use geometric mean for ratio data. Geometric mean is another way to calculate average, which
is useful in measuring differences in time.
How many Participants do you need?

The number of participants needed for a research depends on the goals of your research and
your tolerance for a margin of error. Generally, you need less participant in the first stages of
the design and development, while you need more participant in the later stages to find
remaining issues.

For a quantitative research, what you need to consider is how much statistical errors you can
tolerate. When you do the research with fewer participants, your data tend to contain more
statistical errors. When you do the research with many participants, your data tend to be closer
to the true population. That’s why confidence interval is important. Confidence interval is an
estimate of a range of values that includes the true population value for a statistic, such as a
mean. You decide what level of confidence you need, such as 90% or 95%, and calculate the
confidence interval to show how accurate the measures actually are.

Part B

Students need to apply quant and qual research design methods as per your project scope

Interpret your findings clearly

Conclusion: Your learnings from the lab experiment