IB9BA0 Quantitative Methods for Business

Hypothesis Testing

Professor Jerker Denrell

[email protected]
Professor of Behavioural Science

17/01/2024 1
 Today’s outline
Hypothesis testing
• The basic idea: is a coin biased or not?
• Significance level
• Hypothesis testing for
– Proportions
– Means
– Differences between two means (Video)

17/01/2024 2
 Statistical claims
• 75% of parents think that the IQ of their children is higher
than the average IQ.

• 70% of people apply for the Warwick Business School MBA

Programme only if their post-MBA salary is expected to be
doubled.

• 80% of new business owners think their business has at least

70% chance of success but only 40% think any business like
theirs is likely to succeed.

• Women are better drivers than men.

17/01/2024 3
 Hypothesis testing
Confidence intervals are used to estimate population parameters
such as population means and proportions.

However, managers often want to test if a certain target has been

achieved or if a hypothesis about the population is valid. In
this situation, we apply a tool named Hypothesis Testing to
make inference based on data.

Examples are:
– Is the advertising campaign a success?
– Has a new procedure improved the quality of the product or the
efficiency of a process?
– Are two groups (e.g., males vs females) different with regard to a
certain characteristic?

17/01/2024 4
 An illustration: The biased coin test
Alex thinks a coin may be biased. She throws it 10 times and
gets tail 9 times, i.e., 90%. On the basis of this evidence she
claims that the coin is biased. After all, she says, it is unlikely
that one would observe 9 tails in 10 trials if the coin was
unbiased.

Is she correct? How should we evaluate whether the coin is

unbiased?

We need to compare the outcome we observe (9 tails in 10

trials) with what is likely to happen if the coin was unbiased.

17/01/2024 5
 An illustration: The biased coin test
If the coin was unbiased, i.e., pr(tail)= 0.5, then this
would be the distribution of the number of tails:

Picture of the number of tails when

an unbiased coin is tossed 10 times

Number of tails
Alex observes 9 out 10

So: we reject the

If the coin was
hypothesis that This is an unlikely result:
unbiased, it is very
the coin is Pr( 9 tails or more if
unlikely (1%) that we
unbiased. coin unbiased)
would observe a
We may be = 0.0108 (we can get
result this extreme
wrong, but it is this from the table)
(i.e. 9 or more).
unlikely
17/01/2024 6
 The logic behind the “The biased coin test”
Data observed:
9 out of 10 tails Decision criteria:
When to reject the null
hypothesis?
Null-Hypothesis to be tested:
The coin is unbiased, i.e., Reject the null-hypothesis
pr(tail) = 0.5 if Pr(9 or more if pr(tail) =
0.5) is low.
Evaluation of the Null-hypothesis That is, reject the null
What is the probability of the data hypothesis if the data is
(9 out of 10), if the null hypothesis very unlikely if the null
was true (pr(tail) = 0.5). hypothesis was correct

How low? Depends on how

Actually, we computed: sure you want to be…
Pr(9 or more if pr(tail) = 0.5)
17/01/2024 7
 How sure do you want to be?
Suppose you rejected the hypothesis that a coin was
unbiased whenever you observed at least 9 tails or at
most 1 tail in 10 trials. What is the probability you would
mistakenly claim that an unbiased coin is biased?
Picture of the number of tails when an
unbiased coin is tossed 10 times

It is 0.0108+0.0108 =
0.0216

Here you also reject the Here you reject the hypothesis of
Number of tails an unbiased coin. If the coin is
hypothesis of an unbiased
coin. If the coin is indeed indeed unbiased these outcomes
unbiased these outcomes (1 (9 or more) occur with probability
or less ) occur with 0.0108
probability
17/01/2024 0.0108 8
 How sure do you want to be?
Suppose you rejected the hypothesis that a coin was
unbiased whenever you observed at least 8 tails or at
most 2 tail in 10 trials. What is the probability you would
mistakenly claim that an unbiased coin is biased?
Picture of the number of tails when an
unbiased coin is tossed 10 times

It is 0.0547+0.0547 =
0.1094

Here you also reject the Here you reject the hypothesis of
Number of tails an unbiased coin. If the coin is
hypothesis of an unbiased
coin. If the coin is indeed indeed unbiased these outcomes
unbiased these outcomes (2 (8 or more) occur with probability
or less ) occur with 0.0547
probability
17/01/2024 0.0547 9
 Trade-offs between different errors
You do not say
You say it is biased it is biased

Coin is
biased Correct Error

Coin is Error Correct

unbiased

17/01/2024 10
 Suppose I only say it is biased when I have extreme
data: the number of tails is 10 or zero!
Picture of the number of tails when an
unbiased coin is tossed 10 times

The probability of saying it

is biased when it is actually
unbiased is 0.001 + 0.001 =
0.002

Here you also reject the Number of tails Here you reject the hypothesis of
hypothesis of an unbiased an unbiased coin. If the coin is
coin. If the coin is indeed indeed unbiased this outcome
unbiased this outcome (zero) (10) occurs with probability 0.001
occurs with probability 0.001

17/01/2024 1111
 Suppose I only say it is biased when I have extreme
data: the number of tails is 10 or zero!
You say it You do not say
is biased it is biased

BUT: this error is now

Coin is more likely to occur:
Correct
biased because I almost never
say the coin is biased

This error
Coin is is highly
unlikely Correct
unbiased
(Prob =
0.002)

17/01/2024 12
 Please provide evidence

Drug approval
Approve the drug Do not approve the
as safe drug as safe

Drug is safe

Drug may be is
harmful

17/01/2024 13
 Snake or stick?

Picture of an object that may be a snake

17/01/2024 14
Source: Google images
 Suppose I say the coin is biased whenever I have some
evidence: the number of tails is 7 or more or 3 and less!

Picture of the number of tails when an

unbiased coin is tossed 10 times

The probability of saying it

is biased when it is actually
Picture of an object that unbiased is 0.1719 +
may be a snake
0.1719= 0.3438

Here you also reject the Here you reject the hypothesis of
Number of tails an unbiased coin. If the coin is
hypothesis of an unbiased
coin. If the coin is indeed indeed unbiased these outcomes
unbiased these outcomes (3 (7 or more) occur with probability
or less ) occur with 0.1719
probability 0.1719

17/01/2024 15
 Suppose I say the coin is biased whenever I have some
evidence: the number of tails is 7 or more or 3 and less!
Picture of an object that
may be a snake
You do not say
You say it is biased it is biased
BUT: this
error is
Coin is
Correct unlikely: I
biased
often say the
coin is biased

This error is now

Coin is
likely (Prob = Correct
unbiased
0.3438)

17/01/2024 16
 Important Slide: Logic of Hypothesis testing
• I have some data with a result (fex: proportion tails observed = 9
out of 10)

• I want to test a “Null-hypothesis” (fex, is the probability of tails =

50%)

• I have a decision rule: how strong does the evidence have to be

for me to reject the null-hypothesis?

• I evaluate the null-hypothesis by calculating: what is the

probability that I would get the result I observe, or more
extreme, if the null-hypothesis was true? (fex observe 9 or 10 out
of 10). This is called the “P-value”.

• 17/01/2024
I reject the null-hypothesis if the p-value is low enough. 17
 The training manager’s problem
Elizabeth Field is the training manager of a light engineering firm
which employs a considerable number of skilled machine
operators. The firm is constantly making efforts to improve the
quality of its product, and so recently Elizabeth has introduced a
new training course for workers who have been on the same
machines for a long time. The first group has now completed the
course and returned to normal work, and Elizabeth would like to
assess the effect, if any, that the retraining has had upon the
standard of its work so that she can decide whether to make such
courses a regular event.

Assume that Adam, before the course, had been producing 4% of

defective items; after the course, his performance is monitored
during the production of 400 items, and 14 of them are found to be
defective.

Can we conclude that Elizabeth’s training course has made a

difference to the quality of the products that Adam produces?
17/01/2024 18
 A first analysis
Adam’s defect rate in the past was 4% and his current defect
rate is 14/400 = 3.5%.

We would think that he has improved his performance since his

training course. However, we know that 3.5% is just a sample
result and that his performance may be different from another
sample. What we need is to make a sound inference from the
data. Alternatively, we can ask if there is any strong evidence to
say that Adam has indeed improved his performance. In other
words, is 3.5% different from 4% statistically?

Is 3.5% different from 4%?

17/01/2024 19
4%
 Hypothesis testing: Make hypotheses
• Let p and q be Adam’s defect rate in the past and now,
respectively. Make a null hypothesis NH and an alternative
hypothesis AH, which cover all possibilities, but are mutually
exclusive.

• Null Hypothesis: Adam’s current defect rate is the same as

before: q = p.
• Alternative Hypothesis: Adam’s current defect rate is
different from his defect rate in the past: q ≠ p.

• We need to reject or not to reject NH based on some

statistical evidence.

17/01/2024 20
 What is the probability we would observe
a change of this magnitude (or more)?
• Null-hypothesis: the error rate is unchanged,
still = 4%

• Data / result observed: error rate = 14/400 =

3.5%

• How likely is it that we would observe a

change of this magnitude (or more) if the null-
hypothesis was true?
What is the probability that How to calculate this?
we observe a result this low
in a sample of 400 if the 1. Use the binomial formula?
true proportion is 4%? 2. Or (easier) use the Normal
Approximation and the STEP
formula

17/01/2024 21
Picture of a normal distribution
 What is the probability we would observe
a change of this magnitude (or more)?
• Null-hypothesis: the error rate is unchanged,
still = 4%

• Data / result observed: error rate = 14/400 =

3.5%

• How likely is it that we would observe a

change of this magnitude (or more) if the null-
hypothesis was true? We calculate the “t-statistic” of a
What is the probability that result equal to 0.035
we observe a result this low
t = (new error rate – past error
in a sample of 400 if the rate)/STEP
true proportion is 4%?
=
(0.035-0.04)/sqrt(0.04*(1-0.04)/4
00)
17/01/2024
= -0.005/0.0098 22
= -0.51.
Picture of a normal distribution
 Hypothesis testing: Make a comparison
The distance between q (new error rate) and p (prior error
rate) is about 0.51 STEP, which is much smaller than 2 STEP,
the distance based on the 95% confidence level.

-2 -0.51 0
Null hypothesis:
Picture of a normal distribution
new error rate – old error rate = 0

So, it seems we cannot be very confident that the new error

rate is different from the old (the new error rate is NOT
2STEP away from the old)

17/01/2024 23
 Hypothesis testing: The Significance level

In hypothesis testing, we often use “significance levels”

rather than confidence levels. We ask: can we reject the null-
hypothesis “at a given significance level”.

The 5% significance level is the probability that we

incorrectly reject NH if we follow the following rule: reject
NH if the absolute value of the t-statistic > 2

-2 2
Picture of a normal distribution
Reject the null hypothesis
Reject the null hypothesis
17/01/2024 24
 Hypothesis testing: The Significance level
So: we observed t = -0.51.

Can we reject the null-hypothesis? No, -0.51 is not extreme

enough.

-2 2
Picture of a normal distribution
We observed -0.51.

Answer: “At a significance level of 5% we do not reject the null

hypothesis that the new error rate = old error rate = 4%”. Or,
equivalently, we do not reject the statement that Adam has not
changed his error rate after the training course.
17/01/2024 25
 Important Slide: Framework for conducting
hypothesis tests: Proportions

• Assume q is the sample proportion and p is the proportion we

want to compare q to.
• Make a NULL hypothesis: q = p, and an ALTERNATIVE
hypothesis: q ≠ p.
• Select a significance level α and find out its critical value z.
• Calculate STEP, based on the null hypothesis:
– STEP = sqrt(p(1-p)/n)
• Calculate the test statistic: t = (q - p)/STEP.
• Compare t with the critical value z.
• Reject NH if the absolute value of t is larger than z. Otherwise,
do not reject NH.

17/01/2024 26
 Critical values for two-tailed tests
5% significance level 1% significance level

Two tailed 1.96 (2 approximately) 2.58 (3 approximately)

Pictures of a normal distribution

17/01/2024 27
 How sure do you want to be?
Suppose you rejected the hypothesis that a coin was
unbiased whenever you observed at least 9 tails or at
most 1 tail in 10 trials. What is the probability you would
mistakenly claim that an unbiased coin is biased?
Picture of the number of tails when an
unbiased coin is tossed 10 times

It is 0.0108+0.0108 =
0.0216

Here you also reject the Here you reject the hypothesis of
Number of tails an unbiased coin. If the coin is
hypothesis of an unbiased
coin. If the coin is indeed indeed unbiased these outcomes
unbiased these outcomes (1 (9 or more) occur with probability
or less ) occur with 0.0108
probability
17/01/2024 0.0108 28
 In practice
• The statistical software will produce a p-value and one
compares the p-value with a chosen significance level
α, which is often 0.01 or 0.05.
• If p-value is smaller than α/2 (two-tailed test) one
"rejects the null hypothesis” and the result is said to
be ’statistically significant’.
• P-value is the probability of obtaining a test statistic at
least as extreme as the one that was actually observed,
assuming that the null hypothesis is true, that is,
p-value = p (at least as extreme as the data | NH is true)

17/01/2024 29
 Hypothesis testing: Calculate test statistic
and p-value
The t-statistic was calculated based on the following formula:

t = (q-p)/STEP

= (0.035-0.04)/sqrt(0.04*(1-0.04)/400)
= -0.005/0.0098
= -0.51.

What is the p-value?

17/01/2024 30
 P(Z < -0.51) = P(Z > 0.51)
=0.305

So: p-value = 0.305.

= if the null hypothesis was true we

would observe a result 0.51 STEP
away or more about 30.5% of the
time.

What is the significance level this

should be compared to?

5%, but 2.5% each at “each TAIL”:

Since p-value=0.305 > 0.025

(“two tailed” significance
level), we DO NOT reject the
null hypothesis.
17/01/2024 31
 How does the significance level
affect conclusions?
• For the training manager’s problem, we still do not reject NH
at either the 32% significance level since 0.51 < 1 or the 1%
significance level since 0.51 < 3.

-2 2 Picture of a normal distribution

Picture of a normal distribution
• In general, the larger the significance level is, the more likely
that NH is rejected because an increase of the significance
level leads to a increase of the shaded zone.

17/01/2024 32
 How does the sample size affect conclusions?

• Suppose Adam’s current defect rate 3.5% was calculated

based on a sample of 40,000 items that he produced. Then
test statistic
t = (q-p)/STEP = ?.

• What conclusion do we draw?

17/01/2024 33
 Hypothesis testing: An alternative
approach
• Construct a 95% confidence interval around sample prop.
[q – 2*STEP, q + 2*STEP] = [1.54%, 5.46%].
• Compare the 95% confidence interval with the proportion p
= 4%
• We reach the same conclusion: At the 5% significance level,
we do not reject NH since p=4% is included in the 95%
confidence interval.

3.5%
1.54% 5.46%
Picture of a normal distribution

17/01/2024 34
 Important Slide: Framework for
conducting hypothesis testing: Means

• Assume m is the sample mean and μ is the population

mean.
• Make a NULL hypothesis: m = μ, and an ALTERNATIVE
hypothesis: m ≠ μ.
• Select a significance level α and find out its critical value z.
• Calculate test statistic: t = (m - μ)/STEM.
• STEM = σ / sqrt(n) [σ is often estimated by the sample
standard deviation]
• Compare t with the z-value.
• Reject NH if the absolute value of t is larger than the z-
value. Otherwise, do not reject NH.

17/01/2024 35
 Example Calculations

• A firm has adopted a new the web design during 81 days and
experienced an sales increase of 10 (thousands) pound per
day with a standard deviation of 27.
• Can you reject the null hypothesis that there is no sales
increase, using a significance value of 5%?

17/01/2024 36
 Answer

• From the data, we have n = 81, m = 10, and Stdev is 27.

• Null hypothesis: 0, and Alternative Hypothesis: m ≠ μ.

• STEM = σ / sqrt(n) = 27 / sqrt(81) = 27 / 9 = 3.

• Calculate test statistic:

t = (m – Null Hyp)/STEM = (10 - 0)/3= 3.333 approximately.

• Compare test statistic t and the two-tail z value, equal to 2, at the 5%

significance level.

• Make a conclusion: because 3.33 > 2, statistically there is enough

evidence to say that the daily sales increased
17/01/2024 37
 Watch the Videos about...
• One-tailed test
• Test of difference between means

Do tasks for videos...

Try exercises for Lecture 4

17/01/2024 38
 Next lecture
• About ‘analysis of variance’, which is closely
related to today’s test statistics.

• Do check your understanding by solving the

exercises for lectures 1-4!
• Feel free to attend office hours and arrange
sessions with the TAs if you have any questions
• Enjoy!

17/01/2024 39