Doctoral Comprehensive Exam Research Question

You are employed as a school psychologist in a district that has a broad range of
family income levels. The district just started a Vocational Transition Program for
juniors in High School. The program will be scheduled for one period once a week for
the entire school year. You have been asked to evaluate the effectiveness of this
program.

Describe how you would evaluate the effectiveness of this curriculum. Provide
your answer using sentences and paragraphs using the following headings:

Introduction including the purpose,

Research Questions to be answered,
Research/Evaluation Design to be used and why this design was chosen,
Sample and Population to be used,
Variables and Measures to be used and why these measures were chosen,
Procedures you will use,
Data Analysis proposed and why this method of data analysis was chosen,
Ethical Considerations, e.g. the need for Parental/Guardian permission,
Treatment appropriateness, etc.,
Strengths of your research/ program evaluation plan, and
Weaknesses of your research/ program evaluation plan.

Comment from the EDSP faculty

The doctoral program in School Psychology at IUP prepares students to work in a

number of settings ranging from serving as practitioners in schools to teaching at
the university level. Since the majority of our students choose to continue as
practitioners, our emphasis in the area of research is largely on applied research.
Even if our graduates continue to work in the schools, however, we do expect that
they will have sufficient competence to assist in school-based research and
evaluation even in areas outside the normal domains of a school psychologist.
Additionally this is a skill emphasized in the 2010 NASP Standards, reflecting the
importance of this competency regardless of employment location (e.g., academia,
public or private practice).

It should not matter whether you have a significant level of expertise in what is
being evaluated as long as you have a good understanding of how research design,
sampling, measurement, and statistics are connected in an applied setting and you
have sufficient understanding of how you can work with others to develop practical
solutions to evaluation problems and applied research questions.

In the sample response to the question posed, we provide you with some notes that
should assist you in preparing a response to any other similar question. In
addition, a reading list is provided that will help you prepare for this portion of
your comprehensive exam.

1
 Sample Response Notes
Introduction
As a school psychologist, I have been The introduction should primarily
asked to assist the district in set the stage for what will
evaluating a newly implemented follow. It should be reasonably
Vocational Transition Program for brief, but should make it clear
juniors in the high school. In order that you understand the purpose
for me to provide adequate assistance of the task.
in this project, it will be important
for me to work with a team of
district personnel to identify the
criteria for success as well as ways
to collect data regarding the level
of program implementation (fidelity)
and the level to which the various
criteria for success have been met.
Any approach to this evaluation will
need to be realistic and manageable
and also must meet the needs of the
district.

2
 Sample Response Notes
Research Questions
Basically, the primary impetus for my In any program evaluation there
involvement is to assist in the are several ways to approach the
evaluation of the newly implemented basic research question or
Vocational Transition Program (VTP) questions to be addressed. There
for high school juniors. However, is one overarching point,
many more specific questions could be however, and that is the basic
asked and my job would be to help the question of whether the program
district decide which questions are was effective. This is
of most importance and then to help complicated by the different ways
in designing an evaluation model that in which effectiveness can be
addresses these questions. measured and you, for this
question, need to 1) make it
One strategy I could use in the clear that you understand these
beginning would be to create an differences and 2) pick two or
evaluation committee that would help three upon which you will build
oversee the process. An early task the rest of your response to this
would be to help the district comprehensive exam question.
identify exactly what constitutes
effective. That is, what are the
criteria the district wants to use in
determining whether the program is
effective? Identifying these
criteria is important because it will
be directly related to the measures I
will have to collect and,
potentially, the design of the study
as well as the statistical procedures
to be employed.

For example, I would need to know

what successful vocational
transition means to the district.
Does it mean that current juniors who
take the program have a particular
set of skills? Does it mean that the
students who take the program will be
able to get employment during the
summer between their junior and
senior years? Are the goals more
long-range and related to finding
work after graduation?

Another set of issues that would need

to be addressed have to do with the
audience for the program. Are there
certain sub-groups for whom this
program is particularly designed? Is
it intended for special populations
or all juniors? Is it expected to be
particularly effective for boys more
than girls?

3
In addition to these basic questions,
I would need to know whether there
are other areas of particular
interest or concern. For example, is
the district interested in knowing
how satisfied the teachers are in
delivering the program? Is cost of
the curriculum and its implementation
to be monitored and evaluated? Are
there issues of fidelity that need to
be evaluated or is fidelity simply a
factor that needs to be considered in
designing the evaluation?

Finally, it would be important to

know who the district views to be the
stakeholders. Are we to assess the
satisfaction by students, teachers,
parents, community personnel, agency
personnel, or others?

Once I have had the opportunity to

hold these discussions and involve
the appropriate persons, I would be
able to formulate specific questions
around which the program evaluation
should be designed. For the sake of
this exam, I will identify the
following three questions as coming
from the above process:

1. Do juniors who participate in

the program acquire skills X,
Y, and Z?
2. Are teachers who teach the
curriculum satisfied with the
manner with which it is
delivered?
3. Are parents of the children who
participate in the program
satisfied?

4
 Sample Response Notes
Research/Evaluation Design

As stated above, I have selected The first decision Id have to

three specific research questions for make in answering this question
the purpose of this examination and is whether to approach the design
it is these three questions that will of this evaluation from a
guide the remainder of my response. quantitative or a qualitative
approach. A mixed-method
Understanding the three questions is approach is also possible and
important to selecting an appropriate might be considered. Each of
design methodology. The first these is legitimate and a case
question states, Do juniors who can be made for each. My
participate in the program acquire preference in answering is use a
skills X, Y, and Z? This question quantitative approach for the
asks whether the program is effective skills of X, Y and Z; with a
in helping the students achieve the pseudo qualitative, though still
desired skill outcomes of X, Y, and quantitative, approach for the
Z. To answer this question, it would questions dealing with
be important for the design of this satisfaction by teachers and
evaluation design to include some parents.
pre-test data so that it can be
determined whether after the program Ultimately, the choice of the
occurred, there were any changes in method for the quantitative
these skills. However, it will also aspect was made to help minimize
be necessary to consider whether some of the threats to internal
maturation alone might have caused validity.
these changes in skills, so it may be
necessary to consider a control group
to compare against.

The second question states, Are

teachers who teach the curriculum
satisfied with the manner with which
it is delivered? and the third
question states, Are parents of the
children who participate in the
program satisfied? Both of these
questions might best be answered
through a survey of teachers and
parents.

Therefore, I would suggest an

evaluation model that would be
considered a combination of causal-
comparative (ex post facto) for
collecting student data and
descriptive for collecting parent and
teacher data. The general model for
the student data might be:

O1 X O2

O1 C O2

This would include identifying two

5
groups of students, X (treatment
those participating in the
curriculum) and C (control those
NOT participating in the curriculum).
Prior to the start of the program,
students in both groups would be pre-
tested on the three dependent
variables of X, Y, and Z
(Observations at time 1 - O1) and
after the treatment, the groups would
again be tested on the three
dependent variables of X, Y, and Z
(Observations at time 2 O2).

In addition, I would develop a survey

form for collecting satisfaction data
from the teachers who participated in
the delivery of the program and a
second survey form for collecting
satisfaction data from the parents of
the students who participated in the
program.

If possible, I would like to use a

delivery method such as Qualtrics for
my teachers (as I am confident that
they all have access to computers in
the school) and I would probably send
a mailed survey to the parents,
though it may be possible to collect
the data in person at some point if
the parents are all at the school at
the same time.

Sample Response Notes

Sample and Population

Since this is a program evaluation, As with other aspects of the

the primary population of interest answer to this question, there
can best be described as future are a number of approaches that
students and teachers in this could be taken. One approach
district who may take (students) or might be to consider choose a
deliver (teachers) this Vocational sample by selecting a random
Transition Program. In addition, one number of students participating
of the research questions involves in the curriculum, another might
the satisfaction of parents, so be to select all students
parents, also, need to be participating in the curriculum
stakeholders. In other words, the the decision as to what the
district has implemented a program population might be could include
that it would like to continue to use future students, teachers, and
into the future. Therefore, the parents in the district or it
population would include those could be other students, parents
persons who may be involved in this and teachers similar to those in

6
curriculum within the district. One my district.
might argue that, from a research
perspective, a broader population
could include other students,
parents, and teachers (district
personnel) who are similar to those
in the district I am employed. If I
were to set this up as a research
project, per se, I might be
interested in making generalizations
to a larger population beyond my own
district. However, because I am
primarily interested in the
effectiveness of this curriculum in
my own district, I will define the
population accordingly. One positive
outcome of defining my population in
this manner is that issues of
external validity are less of a
concern since, barring any major
shifts in the demographics of my
district, my sample should be quite
representative of this population.

The sample, then, would be the 11th

graders who are enrolled in the VTP,
the parents of these students, and
the teachers who are responsible for
delivering the program.

Sample Response Notes

Variables and Measures

For the purpose of this response, I There are basically five (5)
will define the three dependent dependent variables in this
variables X, Y and Z as: evaluation. These include the
skills (X, Y, and Z) as well as
X: Holland Self-Directed Search the attitude of teachers and the
(HSDS) which identifies personality attitude of the parents.
traits that align to different career
choices. If the skills (X, Y, and Z) are
skills/behaviors that are
Y: Career Occupational Preference measured by some standardized
System (COPS) Interest Inventory instrument, then the response
which is completed by the student and might include how youd go about
identifies broad occupational / identifying those instruments.
vocational areas that align with Alternatively, if you
their reported interests. specifically identify in your
response what those skills are
Z: Career Aptitude Placement Survey and if you know how those skills
(CAPS) which is an assessment of might be measured, your response
reading, writing, mathematics, could include that level of
science, social studies, etc. detail.

7
knowledge and skill to see if the
student has the pre-requisite skills Similarly, if attitudinal
that would predict success (provided instruments for measuring
additional training) in the satisfaction by teachers and
identified broad interest areas. parents already exist, you can
cite those. If they do not
The satisfaction of the teachers in exist, you could discuss in your
developing the curriculum and the answer how you would go about
satisfaction of the parents would be creating such instruments and how
measured by surveys I would create. you would determine the validity
I would identify a group of teachers and reliability of the
who are not directly involved in the instruments.
delivery of the program I am
evaluating in my own district but Just a note about the instruments
who are familiar with such programs, it may well be that these
in general, from another district or particular instruments and the
building in my district. I would construct they purport to assess
then create sample items and might not measure traits that
distribute them to these teachers to would be expected to change over
seek their input into the content the course of time this
validity of the items. I would also evaluation would be conducted.
distribute the test instruments to a They are listed here only as
group of volunteers to determine examples of how a response could
reliability using an internal be crafted.
reliability method such as split half
or one of the Kuder-Richardson
formulas.

Since this is a program evaluation,

and not a research project for
publication, I would most likely not
be as concerned about the reliability
of the instruments as I would be
about the utility of the data in
answering the original evaluation
questions.

Sample Response Notes

Procedures
For this response, I identified three The response to this section is
possible questions to be addressed in somewhat dependent upon the way
this evaluation. These are: in which earlier portions of the
1. Do juniors who participate in question was answered.
the program acquire skills X, Therefore, the procedures need to
Y, and Z? be reasonably specific to the
2. Are teachers who teach the three evaluation questions
curriculum satisfied with the identified above.
manner with which it is
delivered?
3. Are parents of the children who
participate in the program
satisfied?

Consequently, the design or

8
procedures to be employed need to
allow for me to answer these three
questions.

The first question asks whether the

program was successful in helping
students acquire three skills (X, Y,
and Z). To investigate this, I would
recommend that the students be given
a pre-test and a post-test for each
of the three skills.

The remaining two questions deal with

participant (teachers and parents)
satisfaction. I would work with the
personnel in the school to develop
survey instruments and collect data
from participants in that manner.
These surveys would be collected
either in paper/pencil, electronic,
or face-to-face (focus group) formats
depending on what happens to work
best for the participants.

In terms of designs, the first

question would use what is typically
called an ex post facto non-
experimental design. The response to
the second and third questions would
utilize what is typically called
descriptive or survey research.

In addition to collecting the data on

the three outcome questions, I would
make sure that an appropriate measure
of program fidelity was utilized in
order to document that the program
being evaluated was, in fact,
implemented properly. This
additional implementation fidelity
data would also be useful in
documenting any difficulties that may
have been encountered in delivering
the program.

Sample Response Notes

Data Analysis
The statistical analysis would be The response to the section on
dependent on the specific evaluation data analysis is very highly
questions and the type of data dependent upon the type of data
collected. The first evaluation collected. As such, there are
question for which pre and post data many different approaches to this
were collected for the three skill response and it is important to

9
areas (X, Y and Z) would probably be carefully respond to this in light
a paired t-test. The analysis for of those sections.
the remaining two evaluation
questions would most likely be simple
descriptive statistics such as means
and standard deviations.

Sample Response Notes

Ethical Considerations
As a School Psychologist I must take No notes are provided for this
very seriously my obligations portion of the response as it is
regarding ethical practice(s). assumed that ethical
considerations is an area that
For the purpose of this question, I every student in our program
am going to assume that all the data understands.
collected are not normally and
routinely collected by the district
and that the instruments employed
were selected specifically for the
purpose of this evaluation.
Consequently, I cannot claim that I
am simply using routinely collected
archival data. On the other hand, I
am also not conducting a research
project, per se. Rather, I am
conducting an evaluation and schools
do have the right to evaluate their
effectiveness.

In this particular case I am also not

withholding a valuable treatment from
those in need of that treatment in
order to evaluate it. Therefore, I
do not have to worry about that
aspect of my ethical role.

Given these, I would still advise the

district to inform the parents and
teachers that an evaluation was to be
conducted. I would make it clear
that all participants knew and
understood that THEY were not being
evaluated, that data would be
anonymous (when possible) and
confidential for all, and that
participation was voluntary.

Increasingly, and especially among

larger school districts, districts
convene an institutional review board
(IRB) which approves internal (and
external) requests to conduct
research. Assuming my district has

10
such a committee, I would seek
approval from this committee before
approaching prospective participants
and gathering any data to be used in
the program evaluation.

Sample Response Notes

Strengths
I think one of the strengths of this Strengths typically would be ways
approach to the evaluation is that it in which internal and external
tries to build ownership on the part validity are enhanced.
of the district and does not simply
employ me, as the School Psychologist,
to come in and collect data. The
identification of the evaluation
questions and the design of the
evaluation, itself, would be planned
with the appropriate school personnel.

A second strength would be that part

of the design is to monitor program
fidelity. Without some measure of
program fidelity, it would not be
possible to conclude whether the
program was effective.

Sample Response Notes

Weaknesses

As with any kind of program Weaknesses would typically be

evaluation, it is possible that those things that limit internal
students and other participants try to or external validity.
do better simply because they are
being evaluated and know they are
being evaluated. This would be a
threat to the internal validity of the
evaluation. Program satisfaction by
parents and teachers is very
subjective and may not be very
reliable.

11