Peace Bridges Organization

Pretest and Posttest Design

Guideline for Internal Use

I. Purpose

Pre- and post-tests are used to measure knowledge gained from participating in a training course.
The pre-test is a set of questions given to participants before the training begins in order to
determine their knowledge level of the course content. After the completion of the course,
participants are given a post-test to answer the same set of questions, or a set of questions of
comparable difficulty. Comparing participants’ post-test scores to their pre-test scores enables you
to see whether the training was successful in increasing participant knowledge of the training
content.1

Pre-test/post-test comparisons allow assessment of a pedagogical or technological

intervention/training by determining differences in learning outcomes that occur between two points
in time – before and after the training. The comparisons will assess changes in knowledge, skills
and attitudes as a direct result of the training. This is a common evaluative tool in training and
education as its implementation is simple and its analysis does not require advanced statistical
analysis.2

II. Using the Pre- and Post-Test

Pre- and post-test scores provide information on whether or not participants have learned from the
training. In addition, a well-designed pre- and posttest can help trainers understand which concepts
or competencies were well taught during the training and which ones need additional time, or need
to be covered using alternative methods.

Pre- and post-tests may not be the best tools to use for every training. For example, if a course is
highly interactive with ample opportunity to assess participants’ knowledge and comfort with the
training content, then pre- and post-tests may not be worth the time to administer and analyze. Very
short courses, lasting a day or so, may not merit the time requirements of administering and
analyzing pre- and post-test data. In addition, pre- and posttests for trainings focused on
relationship-building skills, e.g., training on effective counseling skills, may be difficult to create
and score. An alternate method of evaluating learning, such as the observation of skills
demonstrated in a role play, might be more appropriate in this situation.

When deciding whether or not to take the time to do both a pre- and a post-test, consider first what
you most want to learn about your training. If you want to understand exactly what knowledge can
be credited to the training itself, using a pre- and post-test methodology is important. If, instead, you
only need to know whether participants can demonstrate content knowledge or skills by the end of
the training, a pre-test is not necessary. By administering a test only at the end of training, you will
not be able to show knowledge gains, but you will be able to assess the extent to which participants
understand the course material. Remember, one of the limitations of any test of knowledge
administered immediately after training is that it will not tell you what people will remember one
week or one year after the training, nor whether they will apply what they learned in their work.
1
http://www.go2itech.org/resources/technical-implementation-guides/TIG2.GuidelinesTesting.pdf/view?
searchterm=pre-%20and%20post-test
2
http://horteval.weebly.com/pre-test-post-test.html

Page 1 of 9
III. Developing a Pre- and Post-Test

Tests are instruments or tools used to measure change. If the instrument itself is faulty, it cannot
accurately measure changes in knowledge. A valid and reliable pre- and post-test must be made up
of well-written and clear questions. The following are some tips for creating good questions:

Create questions that focus on the primary course objectives. Try to develop at least one question
for each course objective. This will ensure that you are asking participants to demonstrate their
knowledge of what course developers determined are the most important concepts to learn across
the entire course. You can go one step further by asking yourself, “What are the 10 most important
things in this course that the participants needs to know about Peace KASH?” Then create your
questions from this list of 10 concepts, facts, or skills.

Do not create questions that demand the memorization of extraneous (i.e., picky) detail. Participants
should not be tested on whether they remember a word or phrase but rather on whether they have
learned important concepts and related facts.

Only include questions to which there were clear answers provided during the course. Do not test
participants on concepts or knowledge that were not sufficiently covered in the course. If there are
important concepts you think should be covered in the course but weren’t, integrate this information
into your own evaluation of the training and recommend it be included the next time the course is
taught.

Develop a test that will take between 10-25 minutes to complete. The amount of time spent on pre-
and post-tests should vary depending on the length of the overall training course and the type of
questions asked. It is reasonable that a test covering a week-long training would be longer than a
test covering a two-day training. Open-ended essay or short-answer questions will tend to take
longer than multiple-choice tests with the same number of questions.

IV. Tips for Creating Questions

There are a variety of question types – open-ended, true/false, multiple-choice, and case-based –
that can be used in your test. Having a variety of knowledge questions can help make a test more
rigorous and interesting. Open-ended questions require participants to use their own words to
respond to a question or comment on a particular scenario. Multiple-choice questions ask
respondents to select among several possible answers. True/false questions ask respondents to
consider only two possible choices. You can ask respondents to demonstrate more specific, detail-
oriented learning with multiple-choice questions than with true/false questions.

If you are designing a test for people who do not speak English as a first language, avoid changing
your question type too many times throughout the course of the test. Participants may get confused
trying to interpret new directions, which distracts from the purpose of the test.

A. Creating Open-Ended Questions

Create questions that are specific enough so that participants clearly understand the scope of the
question. Example: “Explain how HIV-infected children are likely to suffer.” This question is
overly broad. Participants might wonder whether “suffering” refers to abandonment, physical
weakness, or other complications, or a multitude of other types of suffering. “List 3 reasons why
HIV- infected children are more likely to suffer from failure to thrive.” This question specifically
asks participants to address “failure to thrive,” a clinical condition well-covered in the training
session.

Page 2 of 9
 B. Creating True/False Questions

Construct questions that are simply worded, to the point, and unambiguous. Simple sentences are
straightforward and have fewer words than more complex, multiphrase sentences. Vocabulary that
can be interpreted in different ways makes it much more difficult for respondents to answer.

Example: “There are many ways a person can become infected with HIV” uses a word
(many) that can be interpreted in perhaps ten different ways. A better question would be one
that focused on a single mode of transmission: “An individual can become infected with
HIV through a needlestick.”

Stay away from conjunctions such as “and,” “but,” “except,” and “or.” These words imply a
second idea or concept and can be confusing when respondents are answering True/False questions.

Example: The true/false question stem, “HIV can be transmitted during intercourse but only
if the individuals are not using a condom,” is problematic. Although the question appears to
be true, HIV can be transmitted even if individuals are using a condom during intercourse.
The “but” provides too much potential for ambiguity and room for confusion in the
respondent.

C. Creating Multiple-Choice Questions

Develop responses that are substantively distinct from one another. Answers in a multiple-choice
question that are too similar do not provide a respondent with a clear choice. Such questions can
end up testing their ability to make distinctions in spelling or definition instead of making important
discerning choices among crucial concepts in HIV.

Example: Which of the following is the name for the ARV drug abbreviated as ABC?
Abacab
Abacavan
Abacavir
Abracadabra

Although it might be important for participants in an ARV course to know that Abacavir is the
name for ABC, these responses are more about how well they can distinguish slight variations in
spelling. A better selection of responses would be:
Abacavir
Amprenavir
Acyclovir
Amphotericin B

Develop “incorrect” responses that are potentially plausible but clearly wrong. Even your most
knowledgeable learners should not find the correct answer extremely obvious; respondents should
be presented with a selection of answers that they must consider carefully. In the example above,
the correct response in the first list of responses was too obvious for most English speakers. Among
those who speak English as a second or third language, it may be merely a spelling or vocabulary
test.

Make the multiple-choice question text longer than the text of the answers. The majority of
information should be in the question, not the answers. Participants should not be overwhelmed
with words when attempting to answer the question correctly.

Page 3 of 9
Review your questions and answers for usability. Cover up the answers and look at the question.
Someone knowledgeable of the course content should be able to answer the question without
looking at the answers. If possible, ask a second colleague knowledgeable of the course content to
take the test and see how he/she answers the questions. If they have problems answering, then
chances are that your questions are not specific enough.

D. Creating Likert Scale Questions

Statistics: A method of ascribing quantitative value to qualitative data, to make it amenable

to statistical analysis. A numerical value is assigned to each potential choice and a mean figure for
all the responses is computed at the end of the evaluation or survey.

Used mainly in training course evaluations and market surveys, Likert scales usually have five

potential choices (strongly agree, agree, neutral, disagree, strongly disagree) but sometimes go up to
ten or more. The final average score represents overall level of accomplishment or attitude toward
the subject matter. Named after its inventor, the US organizational-behavior psychologist Dr.
Rensis Likert (1903-81).3

AGREEMENT 

 Strongly Agree   Agree Strongly   Agree   Agree 

 Agree   Agree Moderately   Disagree  Undecided 
 Undecided   Agree Slightly   Disagree
 Disagree   Disagree Slightly 
 Strongly Disagree  Disagree Moderately 
 Disagree Strongly

 Agree Very Strongly   Yes   Completely Agree   Disagree

 Agree Strongly   No  Mostly Agree  Strongly 
 Agree   Slightly Agree   Disagree 
 Disagree   Slightly Disagree   Tend to Disagree 
 Disagree Strongly   Mostly Disagree  Tend to Agree 
 Disagree Very  Completely  Agree 
Strongly Disagree  Agree Strongly

FREQUENCY 
 

 Very Frequently  Always   Always   Almost Always 

 Frequently  Very Frequently  Usually   To a Considerable
 Occasionally  Occasionally   About Half the Degree 
 Rarely   Rarely  Time   Occasionally
 Very Rarely  Very Rarely   Seldom   Seldom

3
Read more: http://www.businessdictionary.com/definition/Likert-scale.html#ixzz2sZUsSkJv

Page 4 of 9
  Never  Never  Never

 A Great Deal   Often   Always 

 Much   Sometimes   Very Often 
 Somewhat   Seldom   Sometimes 
 Little   Never  Rarely 
 Never  Never

IMPORTANCE 

 Very Important   Very Important   Very Important  Very Important 

 Important   Moderately  Important  Important 
 Moderately Important  Important   Neither Important  Moderately
 Of Little Importance   Unimportant or Unimportant Important 
 Unimportant  Unimportant  Of Little
 Very Unimportant Importance 
 Unimportant

QUALITY 

 Very Good   Extremely Poor   Good 

 Good   Below Average   Fair 
 Barely Acceptable   Average   Poor
 Poor   Above Average 
 Very Poor  Excellent

LIKELIHOOD 

 Like Me   To a Great Extent   True 

 Unlike Me  Somewhat   False
 Very Little 
 Not at All

 Definitely   Almost Always True   True of Myself 

 Very Probably   Usually True   Mostly True of Myself 
 Probably   Often True   About Halfway True of
 Possibly   Occasionally True  Myself 
 Probably Not   Sometimes But  Slightly True Of Myself 
Page 5 of 9
  Very Probably Not Infrequently True   Not at All True of
 Usually Not True  Myself
 Almost Never True

E. Combining Methods: Creating Case-Based Questions

A case-based question presents a short case study and then poses questions based on the
information available in the case study. Case-based questions require participants to apply discrete
facts or bits of knowledge to a situation they will likely encounter in their workplace.

Case-based questions can be true/false, multiple choice, or short answer. Short-answer questions
will provide you with a better sense of the participants’ knowledge and ability to apply concepts
learned in the training course. The drawback to short-answer questions is that they take more time
to grade.

Example of a Short Answer Case-Based Question

Mrs. S. is a 34 year-old woman who was referred from the VCT center after testing HIV
positive three weeks ago. Her husband recently died from cryptococcal meningitis;
during his hospitalization he tested positive for HIV, causing her to pursue testing.

Four months ago, Mrs. S. was diagnosed with tuberculosis. Upon her diagnosis, she
began a four-drug treatment for TB, and is currently taking isoniazid and ethambutol.
HIV testing was recommended and refused at the time of TB diagnosis. Other than TB
treatment, she is taking no medications, and until her TB diagnosis her medical history
had been normal. She is allergic to trimethoprim/sulfamethoxazole (Bactrim) which
causes a diffuse erythematous maculopapular rash.
On physical exam, she is a thin, well-developed woman in no acute distress. A head,
eye, ear, nose, and throat (HEENT) exam reveals candida on the tongue and buccal
mucosa. Heart, lung, and abdomen exam are normal. Laboratory test data show:

1. For this patient, what evidence (history, physical examination, and/or laboratory
data) supports beginning antiretroviral therapy (ART)?

2. Do you identify any barriers to good ARV adherence? If so, what would you
recommend to address the barrier(s)?

3. What antiretroviral medication(s) would you initiate?

4. What toxicities could be associated with the antiretroviral medication(s) you chose?

Page 6 of 9
Case questions have a case stem, a lead-in question, and possible responses (or a
space for a short answer reply). The case should not give away the answer, nor should
the answer be obvious among the possible options.

Example 1: A 60-year-old man presents with progressive weakness of arms and legs.
He reports difficulty climbing stairs or combing his hair. He also has difficulty
swallowing, but he has no visual complaints. On physical examination, you note a
maculopapular eruption on the eyelids, nose, cheeks, and knuckles. Joint examination
is normal. What is the most likely diagnosis?
Dermatomyositis
Myasthenia gravis
Polymyalgia rheumatica
Rheumatoid arthritis
Systemic lupus erythematosus

Example 2: In this example, a case is presented and is followed by several multiple-

choice questions.

Case Study: John

John is a recently diagnosed HIV-positive male who came to clinic for his initial visit.
His initial laboratory studies show the following: Hgb of 9, CD4 count of 199. On exam
you notice white plaque on his tongue and the roof of his mouth. He has a history of
tuberculosis 5 years ago, which was effectively treated. Otherwise, he seems in good
health.

The following questions have only ONE right answer. Circle the letter of each correct
answer.

1. At what WHO stage of HIV disease is John?

a. Stage 4
b. Stage 3
c. Stage 2
d. Stage 1

2. Is John eligible for antiretroviral medication?

a. Yes
b. No

3. What is the likely diagnosis of the white plaque on John’s tongue?

a. Oral hairy leukoplakia
b. Kaposi’s sarcoma
c. Oral candidiasis (thrush)
d. Lichen sclerosis

V. Validating Pre- and Post-Tests

All pre- and post-tests must be validated before they are considered a reliable data collection tool.
If participants get a question wrong, it should be because of lack of knowledge, not because a
participant interpreted the question differently than it was intended because the question was poorly
written and had more than one correct answer, or because the question addressed content that was
not taught in the course. When a participant gets a question correct, it should be a result of
knowledge in that subject area, not because the incorrect answers were so implausible that it was
easy to guess the correct answer.

Page 7 of 9
As the first step in the validation process, ask four local staff to take the test. Ask them to mark any
questions that were unclear to them when they were taking the test. Have staff discuss with you
their answer to the questions, ensuring that their understanding of the test questions was the same as
what was intended. Although staff may not be representative of the participants who will take the
test, this is a good first step for clarifying questions and responses before you give the test to a
group of training participants. Make adjustments to the questions based on the results of this
“internal pre-testing” with staff before administering to training participants.

The most important step of the validation process takes place with the participants themselves.
After administering the post-test to training participants, review the answers as a group. Ask
participants to explain their answers to the questions to better understand how they were
interpreting the questions. It should be clear from the discussion which questions were confusing to
participants and which ones were clearly written. For questions answered incorrectly, the discussion
should help to determine whether the question was confusing and participants actually understand
the concept being tested, or whether the participants did not acquire the intended knowledge for
some reason. Rewrite the test based on feedback and administer the test to another set of
participants to make sure adjustments clarified any confusing questions.

VI. Implementing a Pre- and Post-Test

Once you have validated your test, you are ready to begin using it to collect data about your training
course. The most effective way to use pre- and post-tests is to match each participant’s pre-test and
post-test answers so you can look at an individual’s change in knowledge as well as the knowledge
change of the group as a whole. Since participants may be uncomfortable putting their name on
their tests, use numbers instead of names as identifiers on the pre- and post-tests. Below are
instructions for administering anonymous pre- and post-tests.

Assign random numbers to each participant. Write numbers from 1 to n (where n equals the
number of participants in the training) on pieces of paper. Ask participants to choose one piece of
paper and to copy the number they drew on the back of the first page of their participant packet
handed out at the training.

Administer the pre-test before beginning the training. Hand out pre-tests to participants and ask
them to note their numbers on the part of the test marked “code number.” Explain to participants
that their responses to the test are anonymous — we don’t know what number they drew. Explain
that the purpose of the pre/post test is to evaluate our training and to help us answer the question,
“Did people learn what we wanted them to from the training?”

Ask participants to answer the pre-test questions. They can ask for clarification for any of the
questions that they do not understand, for example, if there is something in the wording of the
question that does not make sense to them or is ambiguous. A designated trainer should note all
such questions or clarifications on a master copy so the test can be improved for the next round of
training. When participants have finished, collect the pre-tests and give them to the training team
lead.

Administer the post-test at the end of the training. Hand out post-tests and ask participants to write
their code numbers on the top. Remind participants that their codes are unknown to the trainers, but
it is important they use their same unique code on both the pre- and post-tests for trainers to assess
the strengths/weaknesses of the training. Ask participants to answer the post-test questions. Again,
people should feel free to ask for clarification on language or interpretation. The designated trainer
should take note of all such questions on a master copy.

Page 8 of 9
When participants are finished, collect post-tests and give them to the team lead. Discuss the
questions and answers to the test with the participants. For both teaching and evaluation purposes, it
is important to go over the correct answers to the post-test with participants while you are in class.
Reviewing the answers as a group provides immediate feedback to participants on concepts they did
not understand and reinforces the learning of correct information. It also helps to clear up
misconceptions before those misconceptions are institutionalized through practice. Lastly,
reviewing the answers in class provides the trainer with important feedback on areas that are still
not clear to participants, and helps identify areas of the curriculum needing adjustment.

Make sure to include time for pre- and post-tests and the discussion of results clearly in your agenda
for the training course. Participants should see the discussion of the post-test results as a part of the
training, not an optional activity that happens after the training course is complete.

VII. Analyzing Pre- and Post-Test Results

The final step is to analyze the results of the pre- and post-tests both by participant and by question.
Looking at the data in both of these ways will help you learn about both the type of participant that
learned the most from the training (e.g., those with high or low pre-existing knowledge) and the
areas of the training that were most effective for the whole group. Using Excel, SPSS, or another
statistical software package to analyze the results is not required, but it will greatly facilitate the
analysis process.

Create a spreadsheet where each row is a single participant (identified by ID number) and where
each question has two columns — one column indicating if the participant correctly answered the
question on the pre-test and one indicating if the participant correctly answered the question on the
post-test.

Look at changes in correct responses by individual. Did each individual’s score increase? Are there
any discernible patterns indicating which participant scores increased the most? For example, in a
multidisciplinary training, did nurses’ scores increase more than doctors’? Did each individual’s
score increase? Did the overall range of scores change for the group between pre- and post-testing?
Think about the knowledge level of the audience your training is targeting. If the training is aimed
too high, those that scored high on the pre-test will show the most increase on the post-test while
those who scored low may show very little increase. Alternatively, if lower scores climb and higher
scores are stable, the training may be aimed too low. Depending on the purpose of the evaluation
information, you may want to see if any knowledge increase was statistically significant.

Next, look at changes in knowledge by question to uncover which parts of the training were most
effective, that is, resulted in the most increase in knowledge. Remember that a lack of change in
knowledge could indicate either a poorly designed test question or a weakness in the curriculum. If
pre-test scores are high, then there will be little room for knowledge gain as measured on the post-
test. If there are questions that multiple participants are missing in both the pre- and post-test,
consider adjustments to the curriculum to strengthen weak or unclear content areas.

Review all the data to make sure you haven’t missed any clues and document any additional
interesting findings. Use the results to make any necessary adjustments to the training.

Page 9 of 9