Hypothesis Testing

Module 2.4: Hypothesis Testing

© University of New South Wales

School of Risk and Actuarial Studies

1/27
 Hypothesis Testing
Introduction

Outline

Introduction

Introduction to Hypothesis Testing

Hypothesis Testing Procedure
i. Hypotheses and Significance Level
Confidence intervals and hypothesis tests
P-values
iii. Drawing a Conclusion
Examples

2/27
 Hypothesis Testing
Introduction

Hypothesis testing
▶ A hypothesis is a statement about a population parameter:
▶ More precisely, the researcher assumes whether the true (or
population) parameter value belongs to a certain subset of a
parameter space. Mathematically, suppose θ ∈ Θ (Θ is the set
of values θ can take), and the researcher wants to assume that
θ ∈ Θ0 , Θ0 ⊂ Θ;
▶ The goal of an hypothesis test is to use the available sample to
decide between two complimentary complimentary hypotheses:
▶ the null hypothesis, denoted as H0 : θ ∈ Θ0 ⊂ Θ;
▶ the alternative hypothesis, denoted as H1 : θ ∈ Θc0 ;
▶ The test of hypothesis can be used to answer questions of the
type:
▶ Does the treatment X increase the survival odds of patients
with a certain disease?
▶ Does smoking affect people life expectancy?
2/27
 Hypothesis Testing
Introduction to Hypothesis Testing

Outline

Introduction

Introduction to Hypothesis Testing

Hypothesis Testing Procedure
i. Hypotheses and Significance Level
Confidence intervals and hypothesis tests
P-values
iii. Drawing a Conclusion
Examples

3/27
 Hypothesis Testing
Introduction to Hypothesis Testing
Hypothesis Testing Procedure

Hypothesis testing procedure

To test a hypothesis, implement the following steps:
1) Define a statistical hypothesis. That is, define a null
hypothesis, H0 , and an alternative hypothesis, H1 , and a
significance level, 0 ≤ α ≤ 1;
2a) Define a test statistic T (X1 , . . . , Xn ) (= T ); and
2b) Determine the rejection (critical) region C ⋆ such that
Pr(T ∈ C ⋆ |H0 ) = α;
3) Calculate the value of the test statistic given observed data,
T (x1 , . . . , xn ). Draw a conclusion as follows. If
T (x1 , . . . , xn ) ∈ C ⋆ , then reject H0 . Otherwise, fail to reject
H0 .

3/27
 Hypothesis Testing
Introduction to Hypothesis Testing
i. Hypotheses and Significance Level

Hypotheses: Simple vs. Composite

We have different types of hypothesis:
- Simple hypotheses: completely specify the probability
distribution, for example H0 : p = 0.5.
- Composite hypotheses: probability distribution not completely
specified, for example, H0 : p > 0.5.

4/27
 Hypothesis Testing
Introduction to Hypothesis Testing
i. Hypotheses and Significance Level

Simple and composite hypotheses

▶ Simple null v.s. simple alternative; Example:
H0 : θ = θ0 v.s. H1 : θ = θ1 , for given constants θ0 and θ1 .
▶ Simple null v.s. composite alternative; Example:
H0 : θ = θ0 v.s. H1 : θ ∈ Θ1 , for given constant θ0 and set Θ1 .
Where usually the alternative is of the form:
- H1 : θ =
̸ θ0 ;
- H1 : θ < θ 0 ;
- H1 : θ > θ 0 .
▶ Composite null v.s. composite alternative; Example:
H0 : θ ∈ Ω0 v.s. H1 : θ ∈ Ω1 , for given sets Ω0 and Ω1 = Ωc0 .
Where usually the null and alternative are of the form:
- H0 : θ ≤ θ0 vs H1 : θ > θ0 ;
- H0 : θ ≥ θ0 vs H1 : θ < θ0 .

5/27
 Hypothesis Testing
Introduction to Hypothesis Testing
i. Hypotheses and Significance Level

The Test Statistic and the Critical Region

▶ A test statistic, T (X1 , . . . , Xn ) = T , is a function of the
sample (i.e. a statistic), X1 , . . . , Xn , which is used in a
hypothesis test to decide whether to reject H0 in favour of H1 .
▶ A critical (rejection) region, C ⋆ , is a set in which the values of
T for which H0 is rejected in favour of H1 is specified.

6/27
 Hypothesis Testing
Introduction to Hypothesis Testing
i. Hypotheses and Significance Level

Type I and Type II Error

▶ Type I error: [Reject H0 | H0 is true], denoted by α, also called the
level of significance of the test:

α = Pr[Reject H0 | H0 is true]

= Pr (T ∈ C ∗ | θ ∈ Θ0 )

▶ Type II error: [Not Reject H0 | H0 is false]

β = Pr[Not Reject H0 | H0 is false]

= / C ∗ | θ ∈ Θc0 )
Pr (T ∈

Its complement π = 1 − β = Pr (T ∈ C ∗ | θ ∈ Θc0 ) is called power

of the test.
7/27
 Hypothesis Testing
Introduction to Hypothesis Testing
i. Hypotheses and Significance Level

Example
An insurance company wants to assess whether the mean value of
the claim amount filed by the policyholders is equal to 500$. On a
sample of 10 insurance policies the insurance company finds that
the average claim amount is equal to 525$. Suppose the insurer
carries out a test

at a level of 5% and that claims follow a
2
Normal µ, 50 distribution.

8/27
 Hypothesis Testing
Introduction to Hypothesis Testing
i. Hypotheses and Significance Level

Solution
We are testing:

H0 : µ = 500 vs H1 : µ ̸= 500

Hence, under H0 , the we have the test statistic T = X , which under the
null hypothesis µ = 500, follows a Normal distribution with mean 500
and variance 502 /10. The critical region at a level of 5% is therefore the
set of values for the statistics,
√ which exceeds
530.99 = 500 + 1.96 × 50/ 10, where
▶ 1.96 is the one-side quantile of the standard Normal distribution
which leaves 2.5% prob. in the tail (the sample mean follows a
Normal distribution under the CLT);
√
▶ 50 is the known standard deviation (which is then divided by 10;
▶ 500 is the mean of the distribution under the null hypothesis.

9/27
 Hypothesis Testing
Introduction to Hypothesis Testing
i. Hypotheses and Significance Level

Confidence Intervals and Hypothesis Tests

In Module 2.2 we learned how to construct confidence intervals for
the unknown parameter θ:
1 Select the level of significance α.
2 Find a pivot: g (X1 , ..., Xn , θ), whose distribution is free of the
parameter of interest θ.
3 From Pr (L ≤ g (X1 , ..., Xn , θ) ≤ U) = 1 − α find L and U.
4 Invert the pivot to find the 100(1 − α)% CI for θ in terms of
g −1 (X1 , ..., Xn , L) and g −1 (X1 , ..., Xn , U).
How can we use this for conducting a hypothesis test?
Use the pivot as a test statistic!

10/27
 Hypothesis Testing
Introduction to Hypothesis Testing
i. Hypotheses and Significance Level

Example
Suppose we have a random sample X1 , X2 , ..., Xn ∼ N(µ, σ 2 ) with
known variance σ 2 , and we would like to test

H0 : µ = µ0 v.s. H1 : µ < µ0 .

and α = 5%. Consider the following test statistic:

X −µ
T = √ ∼ N(0, 1)
σ/ n

What is the critical (rejection) region?

11/27
 Hypothesis Testing
Introduction to Hypothesis Testing
i. Hypotheses and Significance Level

Example
An extreme value of T is evidence against H0 whilst an extremely
low value of T is evidence in support of H1 . Therefore construct
the critical region such that:

Pr (T ∈ C ⋆ |H0 ) = 5%.

Pr (T < c ⋆ |µ = µ0 ) = 5%.
 
X − µ0
Pr √ < z5% |µ = µ0 = 5%.
σ/ n
 
σ
Pr X < µ0 + z5% × √ |µ = µ0 = 5%.
n
Thus reject the null hypothesis if X < µ0 + z5% × √σ
n

12/27
 Hypothesis Testing
Introduction to Hypothesis Testing
i. Hypotheses and Significance Level

Confidence Intervals and Hypothesis Tests

Notice that a confidence interval can be obtained by “inverting” a
hypothesis test, and vice versa.

Suppose you would reject H0 : µ = µ0 in favour of H1 : µ ̸= µ0 if:

X − µ0
√ ≥ z1−α/2 ,
σ/ n

for a level of significance α.

In other words, we do not reject the null hypothesis if:

X − µ0
√ < z1−α/2 .
σ/ n

13/27
 Hypothesis Testing
Introduction to Hypothesis Testing
i. Hypotheses and Significance Level

Confidence Intervals and Hypothesis Tests

▶ Thus, for the CI of µ0 we have:
   
σ σ
X − z1−α/2 · √ < µ0 < X + z1−α/2 · √ ,
n n

which gives also a 100 (1 − α)% confidence interval for µ0 .

▶ We observe that µ0 falls within the confidence interval if and
only if we do not reject the null hypothesis.
▶ In other words, the confidence interval consists of “all the
values of µ0 for which the null hypothesis is not rejected.”

14/27
 Hypothesis Testing
Introduction to Hypothesis Testing
i. Hypotheses and Significance Level

p-value
▶ For T the test statistic, we reject H0 if T ∈ C ∗
▶ An alternative is to use the p-value
▶ The p-value is the smallest value of α for which the null
hypothesis, given the data, will be rejected.
▶ p-value is the probability that a value of the test statistic is as
extreme or more extreme that the actually observed value,
given the null hypothesis is true.

15/27
 Hypothesis Testing
Introduction to Hypothesis Testing
i. Hypotheses and Significance Level

Another example
Suppose X1 , X2 , . . . , Xn is a random sample from N (θ, 1) and that
we wish to test H0 : θ = 0 versus Ha : θ ̸= 0 with constant α as a
level of significance.
▶ Suppose the observed sample mean X turned out to be 3, we
can calculate the p-value:



p-value = 2min(Pr X ≥ 3 |θ = 0 , Pr X ≤ 3 |θ = 0 )
 
∗ X −θ 3−θ
= Pr |Z | = √ ≥ √ |θ=0
σ/ n σ/ n
√
√

= Pr Z ≥ 3 n + Pr Z ≤ −3 n

▶ If p-value ≤ α, reject the null hypothesis H0 .

* since Z is symmetric about 0
16/27
 Hypothesis Testing
Introduction to Hypothesis Testing
iii. Drawing a Conclusion

Accept or Do not Reject H0 ? Do not Reject!

▶ Important note: ‘not rejecting’ H0 does not mean we accept
or prove H0 as true. We may simply lack the evidence against
H0 .
▶ Analogy: in the English Common Law, one is considered
‘innocent unless proven guilty’.

H0 : the accused is innocent v.s. H1 : the accused is guilty

▶ If we do not reject H0 , it does not mean we have proven the

accused is innocent, we may simply lack the evidence to
support a guilty verdict.
▶ Similarly, we do not ‘accept’ H1 if we reject H0 . Rather, the
evidence ‘supports’ H1 .
17/27
 Hypothesis Testing
Introduction to Hypothesis Testing
iii. Drawing a Conclusion

Statistical vs Practical Significance, Strength of Evidence

▶ Statistical significance does not imply practical significance;
sometimes magnitude is more important than statistical
significance.

▶ Rejecting H0 for a small α does not mean H0 is ‘wrong by a

significant amount’; it means that H0 is rejected based on a
procedure that incorrectly rejects with a small probability

18/27
 Hypothesis Testing
Introduction to Hypothesis Testing
Examples

Example: Type I and II Errors, and Power

Suppose X1 , X2 , . . . , Xn is a random sample from N(θ, 1) and that
we wish to test

H0 : θ = 0 versus H1 : θ ̸= 0.

Consider the rejection (or critical) region as:


C = (X1 , X2 , . . . , Xn ) : X > 2

Question: Based on this information, calculate power, type I and II

errors of this test.

19/27
 Hypothesis Testing
Introduction to Hypothesis Testing
Examples

Example: Type I error

We have that the level of significance is:

α = Pr (T ∈ C ⋆ |H0 is true )


= Pr X > 2 |θ = 0



= Pr X > 2 |θ = 0 + Pr X < −2 |θ = 0
√ √
√ √

= Pr n · X > 2 n + Pr n · X < −2 n
∗ √
√

= Pr Z > 2 n + Pr Z < −2 n
√

= 2 · Φ −2 n .
Pn
* using X |H0 = i=1 Xi /n ∼ N(0, n1 ).

20/27
 Hypothesis Testing
Introduction to Hypothesis Testing
Examples

Example: Type I and II Errors, and Power

The power function is equal to the probability that the sample will
fall within the critical region given the alternative hypothesis:

π (θ) = Pr (T ∈ C |θ ∈ H1 )


= Pr X > 2|θ ∈ H1



= Pr X > 2|θ ∈ H1 + Pr X < −2|θ ∈ H1
∗ √
√

= Pr Z > n (2 − θ) + Pr Z < n (−2 − θ)
√
√

= Φ − n (2 − θ) + Φ n (−2 − θ) .
2
* using X = ni=1 Xi /n ∼ N(θ, σn ) (using m.g.f. technique), and
P
dropping the conditional for brevity.

21/27
 Hypothesis Testing
Introduction to Hypothesis Testing
Examples

Example: Type I and II Errors, and Power

Now let us suppose that the true value of θ = 1 or 3, so H0 is false.
What is the power of our test when θ = 1 or 3?
√
√

π(1) = Φ − n + Φ −3 n ,
√
√

π(3) = Φ n + Φ −5 n .

Suppose that n = 5, we have

 √   √ 
π(1) = Φ − 5 + Φ −3 5 = 0.01267366
√   √ 
π(3) = Φ 5 + Φ −5 5 = 0.9873263.

At θ = 1, we have a huge type II error. At θ = 3, the type II error

is significantly smaller.
22/27
 Hypothesis Testing
Introduction to Hypothesis Testing
Examples

Example: Power Function

Suppose X1 , X2 , . . . , Xn is a random sample from N(θ, 1) and that
we wish to test: H0 : µ = µ0 v.s. H1 : µ > µ0 .
We know: a test at significance level α would reject H0 if:

X − µ0
T = √ ≥ z1−α
σ/ n

Exercise: Find the power of this test at any value µ > µ0 .

Solution:

23/27
 Hypothesis Testing
Introduction to Hypothesis Testing
Examples

Example: Power Function

Suppose X1 , X2 , . . . , Xn is a random sample from N(θ, 1) and that
we wish to test: H0 : µ = µ0 v.s. H1 : µ > µ0 .
We know: a test at significance level α would reject H0 if:
X − µ0
T = √ ≥ z1−α
σ/ n
Exercise: Find the power of this test at any value µ > µ0 .
Solution: 
X − µ0

π(µ) =Pr √ ≥ z1−α µ > µ0
σ/ n
 
X −µ µ0 − µ
=Pr √ ≥ z1−α + √ µ > µ0
σ/ n σ/ n
 
µ0 − µ
=1 − Φ z1−α + √ .
σ/ n
23/27
 Hypothesis Testing
Introduction to Hypothesis Testing
Examples

Power function
When n increases the power of the test is higher, i.e., more likely to
reject H0 if H1 is correct.

n=20
α n=100
µ0 µ
24/27
 Hypothesis Testing
Introduction to Hypothesis Testing
Examples

Finding the most powerful tests when a single parameter is

unknown (not assessable)
▶ Simple vs simple - Neyman-Pearson Lemma
▶ Otherwise, it can be more challenging; might exist for
one-sided tests, generally doesn’t exist for two-sided tests
(referred to as uniformly most powerful tests, if they exist)

25/27
 Hypothesis Testing
Introduction to Hypothesis Testing
Examples

Quick Questions (recap for home)

▶ What is a statistical hypothesis?

▶ What steps are involved in a hypothesis testing?

▶ How do we choose the Null hypothesis?

▶ What are Type I and Type II errors?

26/27
 Hypothesis Testing
Introduction to Hypothesis Testing
Examples

Quick Questions (recap for home)

▶ What is a power of the test?

▶ What is the confidence/significance level?

▶ What is a critical region?

▶ How are critical region and p-value related?

27/27