Hypothesis Testing

A hypothesis test (or significance test) is the second major method for conducting statistical
inference. Like a confidence interval for estimating a parameter, the hypothesis test is based on
sample evidence and probability, to quantify how plausible a parameter value is while controlling
the chance of an incorrect inference.
In a hypothesis test, hypotheses should be specified beforehand. A hypothesis is a statement about
a population, usually claiming that a population parameter takes a particular numerical value or falls
in a certain range of values. For a categorical variable, the parameter is typically a proportion, while
for a quantitative variable, the parameter is typically a mean.
A hypothesis test has a series of steps.
Step 1: Assumptions
Each hypothesis test makes certain assumptions or has certain conditions under which it applies.
For example, a test assumes that the sample data are obtained using randomisation and the sample
size or the shape of the population distribution meet certain requirements.
Step 2: Hypotheses
Each test has two hypotheses about a population parameter, null hypothesis and alternative
hypothesis.
Null hypothesis is a statement that a parameter takes a particular value. Normally the null hypothesis
is a statement of status quo, e.g., no change, no effect, or no difference in a given situation and is
assumed true until evidence indicates otherwise.
Alternative hypothesis is the opposite of null hypothesis which is a statement that suggests that
there is a significant change or different in a given situation.
The null hypothesis and alternative hypothesis are mutually exclusive.
Step 3: Test statistic
The parameter to which the hypotheses refer has a point estimate. A test statistic describes how
far that point estimate falls from the parameter value given in the null hypothesis. The distance is
measured by the number of standard errors between the point estimate and the parameter.
Step 4: Significance level
The significance level of a hypothesis test is the probability that the observed event could have
occurred by chance or at random. Significance level is denoted as alpha (α). Common value of α =
0.05 (5%), which means a 5% that the observed event occurred by chance, and thus the risk of
rejecting null hypothesis when it is true is also 5%. A significance level of 5% also translates to a
confidence level of 95%.
1
A hypothesis testing is statistically significant if, according to a pre-specified threshold probability
(i.e., significance level), the observed value of the test statistic would be unlikely to occur if the null
hypothesis were true, thus, to reject the null hypothesis. Otherwise, the hypothesis testing is
deemed statistically not significant. That is, to accept the null hypothesis.
Significance level is determined before a test is conducted. Lower significance levels indicate that
you require stronger evidence before you will reject the null hypothesis.
Step 5: Interpretation of test statistic
The decision is made based on the observed value of the test statistic. There are two approaches
on how to derive at that decision: the critical value approach and the p-value approach.
Let’s first look at the Critical value approach. By applying the critical value approach, it is determined,
whether the observed test statistic calculated from sample data is more extreme than a defined
critical value. Critical value at a certain significance level can be thought of as a cut-off point. When
the test statistic is compared to the critical value:
• If the test statistic is more extreme than the critical value, the null hypothesis is rejected.
• If the test statistic is not as extreme as the critical value, the null hypothesis is not rejected.
The critical value is computed based on the given significance level α and the type of probability
distribution of the idealised model. The critical value divides the area under the probability
distribution curve in rejection region(s) and in non-rejection region, as shown in the figure.
• Critical region, aka rejection region, at a certain significant level (α) is a set of values for the test
statistic for which the null hypothesis is rejected. i.e., if the observed test statistic falls in the
critical region, then null hypothesis is rejected and accept alternative hypothesis. The significance
level (α) is the area in the critical region.
• Confidence interval, aka non-rejection region / acceptance region, is a set of values for the test
statistic for which the null hypothesis is accepted. i.e., if the observed test statistic is in the
confidence interval, then we accept null hypothesis and reject alternative hypothesis. As a closely
related concept, confidence level represents the proportion of possible confidence intervals that
contain the true value of the population parameter. Confidence level = (1- α) where α is
significance level.

Now we look at the p-value approach for the interpretation of test statistic. P-value (or probability
value) is the likelihood that the observed results would occur if the null hypothesis were correct, or
the probability that random chance would produce the given results (i.e., results from sample data
occurred by chance). E.g., a p-value of 0.01 means there is only a 1% probability that the results
from an experiment happened by chance. P-value summarises the evidence about null hypothesis.

2
For the p-value approach, the likelihood (p-value) of the value of the test statistic is compared to
the specified significance level (α) of the hypothesis test:
• If the p-value of the experiment result is below a certain pre-determined threshold (i.e.,
significance level), the null hypothesis will be rejected. That also means hypothesis test is
statistically significant. For example, in the figure here, the p-value is the green area to the right
of the test statistic. The test statistic falls in the critical region which is the yellow area, so the p-
value is less than the significance level (α). Therefore, null hypothesis will be rejected.
• A p-value higher than significance level is not statistically significant and indicates strong evidence
for the null hypothesis, which means we will not reject the null hypothesis.
The smaller the p-value, the stronger the evidence against the null hypothesis provided by the
sample.
Step 6: Conclusions
Finally, conclusion must be stated based on the decision whether to reject the null hypothesis, in
the context of the study.
Overall, in every hypothesis test, we are always deciding whether to reject the null hypothesis or to
fail to reject the null hypothesis. In the parlance of law, we are “innocent until proven guilty”.
We should never accept a hypothesis (either null or alternative) because to do so suggests that
there are no other possible avenues or hypotheses that could be considered.