Cochran’s Q Test

I. Definition

Cochran’s Q Test is a non parametric way to find

differences in matched sets of three or more frequencies
or proportions.mIt can be considered to be similar to the
one-way repeated measures ANOVA, but for a
dichotomous rather than a continuous dependent
variable, or as an extension of McNemar's test.
Cochran's Q test is commonly used to analyze
longitudinal study designs, as well as being used when
analyzing participants that have undergone multiple
different trials (a.k.a. treatments/conditions).

When you run a Cochran's Q test, you will either get a statistically significant
result or a non-statistically significant result. If your result is not statistically significant,
it indicates that the percentages/proportions at the different time points or under the
different treatments/conditions are the same in the population. However, if you did get a
statistically significant result, you will probably want to follow up your Cochran's Q test
with a post hoc analysis. You will want to do this because Cochran's Q test is an omnibus
test.

II. Guide to understand

The test is usually used when you have a group of people performing a series of
tasks or getting a set of treatments where the outcome is a group of people performing
a “Success” or “Failure.” Like the binomial distribution, success/failure could mean
yes/no, black/white or any one of a number of success/failure options.

Hypotheses
The null hypothesis for Cochran’s Q is that the proportion of “successes” is equal
for all groups.

The alternate hypothesis is that the proportion is different for at least one group.
The null hypothesis is rejected when the calculated Q critical value is greater than a
critical chi-squared value.
 Cochran’s will tell you if there is a difference, but it won’t tell you where those
differences are. If you reject the null hypothesis (i.e. the test identifies differences),
perform follow up pairwise Cochran’s Q tests to identify the areas which have
differences.

What to remember ?
● You must have one dependent, dichotomous variable. Dichotomous variables are
categorical variables with two categories or levels. The groups/levels must be
mutually exclusive. For example, you could have “Pass” and “Fail”.

● You must have one independent variable with three groups or levels. For
example, you might use three different tests or administer three different
treatments.

● Ideally, use a random sampling method (e.g. simple random sampling).

● You must have a large enough sample size. A rule of thumb is that the number of
subjects, n, multiplied by the number of levels in the independent variable, k, are
greater than or equal to 24.

The Cochran’s Q test and associated multiple comparisons require the following
assumptions:
1. Responses are binary and from k matched samples.
2. The subjects are independent of one another and were selected at random from a
larger population
3. The sample size is sufficiently “large”. (As a rule of thumb, the number of
subjects for which the responses are not all 0’s or 1’s, n, should be ≥ 4 and nk
should be ≥ 24. This assumption is not required for the exact binomial McNemar
test.)

III. Cochran’s Q Test Formula

2 2
𝑄= ( )
(𝑘−1)[ 𝑘•∑𝐺 −(∑𝐺) ]
(𝑘•∑𝐿)−∑𝐿²

Where:

K = number of cases/ treatments

G = sum of each column

L = sum of each row

k-1 = degrees of freedom

Cochran’s Q Test Application

● Two ways

Same qualitative (binary) variable is measured at two or more different times from
the same sample. Two (2) or more comparable qualitative variables are measured
from the same sample (usually at the same time).

● Two Specific Hypotheses

One (1) characterizes the conditions as representing a single population under two or
more different circumstances. Representing comparable variables measured from a
single population.

Sample H0:

H0: The frequencies (or proportions) of responses to the categories of the response
variable is the same across the conditions, for the population represented by the sample.

• To reject H0: is to say that within this population, subjects respond differently
across the conditions.

Ho: Members of this population are assigned into the same categories across the
response variables.

• To reject H0: is to say that cases are categorized differently using the different
response variables.

IV. Steps to Perform Cochran’s Q Test

Cochran’s Q test uses the following null and alternative hypotheses:

Null Hypothesis (H0): The proportion of “successes” is the same in all groups.

Alternative Hypothesis(HA): The proportion of “successes” is different in at least

one of the groups

The test statistic is calculated as:

Where:

K: The number of treatments (or “groups”)

X.j: The column total for the jth treatment

B: The number of blocks

Xi.: The row total for the ith block

N: The grand total

The test statistic T follows a Chi-Square distribution with k-1 degrees of freedom.

If the p-value associated with the test statistic is less than a certain significance
level (like α = .05), we can reject the null hypothesis and conclude that we have sufficient
evidence to say the proportion of “successes” is different in at least one of the groups.

Example: Cochran’s Q Test

Suppose a researcher wants to know if three different studying techniques lead

to different proportions of pass rates among students.

To test this, she recruits 20 students to each take an exam of equal difficulty using three
different studying techniques. The results are shown below:
 To perform Cochran’s Q test, we can use statistical software since it can be
cumbersome to perform by hand.

The code use to create this dataset and perform Cochran’s Q test in the statistical
programming language R:

#load DescTools package

Library(DescTools)

#create dataset

Df <- data.frame(student=rep(1:20, each=3),

Technique=rep(c(‘A’, ‘B’, ‘C’), times=20),

Outcome=c(1, 1, 0, 1, 0, 0, 1, 1, 1, 1, 1, 0, 1, 0, 1,

1, 1, 0, 1, 0, 1, 0, 0, 0, 0, 1, 0, 0, 1, 1,

1, 0, 0, 1, 1, 0, 1, 0, 1, 1, 0, 1, 0, 1, 1,

1, 0, 0, 0, 1, 0, 0, 1, 1, 0, 0, 1, 0, 1, 1))

#perform Cochran’s Q test

CochranQTest(outcome ~ technique| student, data=df)

Cochran’s Q test

Data: outcome and technique and student

Q = 0.33333, df = 2, p-value = 0.8465

From the output of the test we can observe the following:

● The test statistic is 0.333

● The corresponding p-value is 0.8465
 Since this p-value is not less than .05, we fail to reject the null hypothesis.

This means we do not have sufficient evidence to say that the studying technique used
by students leads to different proportions of passing rates.

V. How do you report a Cochran Q Test?

It is important when reporting Cochran's Q test result that you also interpret and
report descriptive statistics in order to get a 'feel' for your data, as well as to provide an
appropriate description of your data. One important measure is that we can report the
number of students to each take an exam of equal difficulty using three different
studying techniques.

If the statistical significance value (i.e., p-value) is less than .05 (i.e., p < .05), you
have a statistically significant result and that we have sufficient evidence to say the
proportion of “successes” is different in at least one of the groups. Alternatively, if p >
.05, you do not have a statistically significant result and the proportion of that we have
sufficient evidence to say the proportion of “successes” is different in at least one of the
groups. In our example, since this p-value is not less than .05, we fail to reject the null
hypothesis, which means we do not have sufficient evidence to say that the studying
technique used by students leads to different proportions of passing rates.

Interpreting the results of Cochran's Q test

The null hypothesis for Cochran’s Q is that the proportion of “successes” is equal
for all groups. The alternate hypothesis is that the proportion is different for at least one
group. The null hypothesis is rejected when the calculated Q critical value is greater than
a critical chi-squared value.

Cochran’s will tell you if there is a difference, but it won’t tell you where those
differences are. If you reject the null hypothesis (i.e., the test identifies differences),
perform follow up pairwise Cochran’s Q tests to identify the areas which have
differences.

Reporting the result of the Cochran Q Test

Based on the statistical software we use. we could report the results of the study as
follows:
 Twenty participants were recruited to take part in study to examine whether
three different studying techniques lead to different proportions of pass rates among
students. Cochran's Q test determined that there was a statistically significant difference
in the proportion of students who passed their exams over time, p > .0005. The
proportion of “successes” is different in at least one of the groups.

VI. Conclusion

In summary, After all is said and done, the Cochran Q test is a test that can
determine if there is a difference in terms of dichotomous dependent variable between
three or more related groups. This is a test that was invented and named after by a
prominent statistician, William Gemmeli Cochran. We can use this test in this example:
to determine whether the proportion of participants who had low self-esteem (as
opposed to high self-esteem) decreased after a series of three counseling sessions (i.e.,
your dependent variable would be "level of self-esteem", which has two categories: "low"
and "high", measured at four time points: "before the first counseling session", "after the
second counseling session", "after the third counseling session" and "after the final
counseling session") (Cochran's Q test using SPSS Statistics, 2018).

Similar to the binomial distribution, success/failure could mean yes/no,

black/white, or any one of a number of success/failure options. The test is typically used
when you have a group of people performing a series of tasks or getting a set of
treatments, where the outcome is a group of people performing a "Success" or "Failure."
The proportion of "successes" for each group must be the same in order for Cochran's Q
to be true.

However, Cochran's C test, a variance outlier test, should not be confused with
Cochran's Q test. Cochran's Q test calls for more than two groups of the same size and
just a binary result (such as success/failure or 1/0), to put it simply in technical terms.
The test evaluates if the percentage of achievements is comparable across groups. It is
frequently used to determine whether the findings of various observers of the same
phenomena are reliable (interobserver variability). While the Cochran’s C test is a single
estimate of a variance (or standard deviation) is compared to a group of variances (or
standard deviations) that are meant to be comparable, and the C test is used to
determine if the single estimate is considerably higher than the group.
References

(2020, April 15). YouTube. Retrieved July 14, 2022, from

https://www.coursehero.com/file/139422324/Cochrans-q-testpptx/

(2020, April 15). YouTube. Retrieved July 14, 2022, from

https://www.statology.org/cochrans-q-test/?%20fbclid=IwAR2ivRlfLKqIxRkGoek

8kA8XISW3ljljars_PLRqi5gQK5jJ7LOqZkGOy%20O4

(2020, April 15). YouTube. Retrieved July 14, 2022, from

https://statistics.laerd.com/spss-tutorials/cochrans-q-test-in-spss-%20statistics.ph

Cochran's Q Test. (2016, July 18). Statistics How To. Retrieved July 14, 2022, from

https://www.statisticshowto.com/cochrans-q-test/

Cochran's Q test in SPSS Statistics - Procedure, output and interpretation of the output using a

relevant example. (n.d.). Laerd Statistics. Retrieved July 14, 2022, from

https://statistics.laerd.com/spss-tutorials/cochrans-q-test-in-spss-statistics.php

What is Cochran's Q Test? (Definition & Example). (2021, January 26). - Statology. Retrieved

July 14, 2022, from https://www.statology.org/cochrans-q-test/

(Cochran's Q test using SPSS Statistics, 2018)

https://statistics.laerd.com/spss-tutorials/cochrans-q-test-in-spss-statistics.php