Yin-Yang Map

Navigating the Knowledge Base

The Yin-Yang Map

The Yin and the Yang of Research

You can use the figure above to find your way through the material in the Knowledge Base.
Click on any part of the figure to move to that topic.
The figure shows one way of structuring the material in the Knowledge Base. The left side of the
figure refers to the theory of research. The right side of the figure refers to the practice of
research.
The yin-yang figure in the center links you to a theoretical introduction to research on the left
and to the practical issue of how we formulate research projects on the right.
The four arrow links on the left describe the four types of validity in research. The idea of
validity provides us with a unifying theory for understanding the criteria for good research. The
four arrow links on the right point to the research practice areas that correspond with each
validity type. For instance, external validity is related to the theory of how we generalize
research results. It's corresponding practice area is sampling methodology which is concerned
with how to draw representative samples so that generalizations are possible.

Foundations
This section provides an overview the major issues in research and in evaluation. This is
probably the best place for you to begin learning about research.

We have to begin somewhere. (Although, if you think about it, the whole idea of hyperlinked
text sort of runs contrary to the notion that there is a single place to begin -- you can begin
anywhere, go anywhere, and leave anytime. Unfortunately, you can only be in one place at a
time and, even less fortunately for you, you happen to be right here right now, so we may as well
consider this a place to begin.) And what better place to begin than an introduction? Here's where
we take care of all the stuff you think you already know, and probably should already know, but
most likely don't know as well as you think you do.
The first thing we have to get straight is the language of research. If we don't, we're going to
have a hard time discussing research.
With the basic terminology under our belts, we can look a little more deeply at some of the
underlying philosophical issues that drive the research endeavor.
We also need to recognize that social research always occurs in a social context. It is a human
endeavor. Therefore, it's important to consider the critical ethical issues that affect the researcher,
research participants, and the research effort generally.
Where do research problems come from? How do we develop a research question? We consider
these issues under conceptualization.
Finally, we look at a specific, and very applied, type of social research known as evaluation
research.
That ought to be enough to get you started. At least it ought to be enough to get you thoroughly
confused. But don't worry, there's stuff that's far more confusing than this yet to come.

Language Of Research
Learning about research is a lot like learning about anything else. To start, you need to learn the
jargon people use, the big controversies they fight over, and the different factions that define the
major players. We'll start by considering five really big multi-syllable words that researchers
sometimes use to describe what they do. We'll only do a few for now, to give you an idea of just
how esoteric the discussion can get (but not enough to cause you to give up in total despair). We
can then take on some of the major issues in research like the types of questions we can ask in a
project, the role of time in research, and the different types of relationships we can estimate.
Then we have to consider defining some basic terms like variable, hypothesis, data, and unit of
analysis. If you're like me, you hate learning vocabulary, so we'll quickly move along to
consideration of two of the major fallacies of research, just to give you an idea of how wrong
even researchers can be if they're not careful (of course, there's always a certainly probability
that they'll be wrong even if they're extremely careful).

Five Big Words

Research involves an eclectic blending of an enormous range of skills and activities. To be a
good social researcher, you have to be able to work well with a wide variety of people,
understand the specific methods used to conduct research, understand the subject that you are
studying, be able to convince someone to give you the funds to study it, stay on track and on
schedule, speak and write persuasively, and on and on.
Here, I want to introduce you to five terms that I think help to describe some of the key aspects
of contemporary social research. (This list is not exhaustive. It's really just the first five terms
that came into my mind when I was thinking about this and thinking about how I might be able
to impress someone with really big/complex words to describe fairly straightforward concepts).
I present the first two terms -- theoretical and empirical -- together because they are often
contrasted with each other. Social research is theoretical, meaning that much of it is concerned
with developing, exploring or testing the theories or ideas that social researchers have about how
the world operates. But it is also empirical, meaning that it is based on observations and
measurements of reality -- on what we perceive of the world around us. You can even think of
most research as a blending of these two terms -- a comparison of our theories about how the
world operates with our observations of its operation.
The next term -- nomothetic -- comes (I think) from the writings of the psychologist Gordon
Allport. Nomothetic refers to laws or rules that pertain to the general case (nomos in Greek) and
is contrasted with the term "idiographic" which refers to laws or rules that relate to individuals
(idios means 'self' or 'characteristic of an individual ' in Greek). In any event, the point here is
that most social research is concerned with the nomothetic -- the general case -- rather than the
individual. We often study individuals, but usually we are interested in generalizing to more than
just the individual.
In our post-positivist view of science, we no longer regard certainty as attainable. Thus, the
fourth big word that describes much contemporary social research is probabilistic, or based on
probabilities. The inferences that we make in social research have probabilities associated with
them -- they are seldom meant to be considered covering laws that pertain to all cases. Part of the
reason we have seen statistics become so dominant in social research is that it allows us to
estimate probabilities for the situations we study.
The last term I want to introduce is causal. You've got to be very careful with this term. Note
that it is spelled causal not casual. You'll really be embarrassed if you write about the "casual
hypothesis" in your study! The term causal means that most social research is interested (at some
point) in looking at cause-effect relationships. This doesn't mean that most studies actually study
cause-effect relationships. There are some studies that simply observe -- for instance, surveys
that seek to describe the percent of people holding a particular opinion. And, there are many
studies that explore relationships -- for example, studies that attempt to see whether there is a
relationship between gender and salary. Probably the vast majority of applied social research
consists of these descriptive and correlational studies. So why am I talking about causal studies?
Because for most social sciences, it is important that we go beyond just looking at the world or
looking at relationships. We would like to be able to change the world, to improve it and
eliminate some of its major problems. If we want to change the world (especially if we want to
do this in an organized, scientific way), we are automatically interested in causal relationships --
ones that tell us how our causes (e.g., programs, treatments) affect the outcomes of interest.

Types of Questions
There are three basic types of questions that research projects can address:
1. Descriptive.When a study is designed primarily to describe what is going on or what
exists. Public opinion polls that seek only to describe the proportion of people who hold
various opinions are primarily descriptive in nature. For instance, if we want to know
what percent of the population would vote for a Democratic or a Republican in the next
presidential election, we are simply interested in describing something.
2. Relational.When a study is designed to look at the relationships between two or more
variables. A public opinion poll that compares what proportion of males and females say
they would vote for a Democratic or a Republican candidate in the next presidential
election is essentially studying the relationship between gender and voting preference.
3. Causal.When a study is designed to determine whether one or more variables (e.g., a
program or treatment variable) causes or affects one or more outcome variables. If we did
a public opinion poll to try to determine whether a recent political advertising campaign
changed voter preferences, we would essentially be studying whether the campaign
(cause) changed the proportion of voters who would vote Democratic or Republican
(effect).
The three question types can be viewed as cumulative. That is, a relational study assumes that
you can first describe (by measuring or observing) each of the variables you are trying to relate.
And, a causal study assumes that you can describe both the cause and effect variables and that
you can show that they are related to each other. Causal studies are probably the most demanding
of the three.

Types of Relationships
A relationship refers to the correspondence between two variables. When we talk about types of
relationships, we can mean that in at least two ways: the nature of the relationship or the pattern
of it.
The Nature of a Relationship
While all relationships tell about the
correspondence between two variables, there is a
special type of relationship that holds that the two
variables are not only in correspondence, but that
one causes the other. This is the key distinction
between a simple correlational relationship and a
causal relationship. A correlational relationship
simply says that two things perform in a
synchronized manner. For instance, we often talk
of a correlation between inflation and
unemployment. When inflation is high,
unemployment also tends to be high. When
inflation is low, unemployment also tends to be
low. The two variables are correlated. But
knowing that two variables are correlated does not tell us whether one causes the other. We
know, for instance, that there is a correlation between the number of roads built in Europe and
the number of children born in the United States. Does that mean that is we want fewer children
in the U.S., we should stop building so many roads in Europe? Or, does it mean that if we don't
have enough roads in Europe, we should encourage U.S. citizens to have more babies? Of course
not. (At least, I hope not). While there is a relationship between the number of roads built and the
number of babies, we don't believe that the relationship is a causal one. This leads to
consideration of what is often termed the third variable problem. In this example, it may be that
there is a third variable that is causing both the building of roads and the birthrate, that is causing
the correlation we observe. For instance, perhaps the general world economy is responsible for
both. When the economy is good more roads are built in Europe and more children are born in
the U.S. The key lesson here is that you have to be careful when you interpret correlations. If you
observe a correlation between the number of hours students use the computer to study and their
grade point averages (with high computer users getting higher grades), you cannot assume that
the relationship is causal: that computer use improves grades. In this case, the third variable
might be socioeconomic status -- richer students who have greater resources at their disposal
tend to both use computers and do better in their grades. It's the resources that drives both use
and grades, not computer use that causes the change in the grade point average.
Patterns of Relationships
We have several terms to describe the major
different types of patterns one might find in a
relationship. First, there is the case of no
relationship at all. If you know the values on one
variable, you don't know anything about the
values on the other. For instance, I suspect that
there is no relationship between the length of the
lifeline on your hand and your grade point average. If I know your GPA, I don't have any idea
how long your lifeline is.
Then, we have the positive relationship. In a positive relationship, high values on one variable
are associated with high values on the other and low values on one are associated with low
values on the other. In this example, we assume an idealized positive relationship between years
of education and the salary one might expect to be making.
On the other hand a negative relationship implies
that high values on one variable are associated
with low values on the other. This is also
sometimes termed an inverse relationship. Here,
we show an idealized negative relationship
between a measure of self esteem and a measure
of paranoia in psychiatric patients.
These are the simplest types of relationships we
might typically estimate in research. But the
pattern of a relationship can be more complex
than this. For instance, the figure on the left shows
a relationship that changes over the range of both
variables, a curvilinear relationship. In this
example, the horizontal axis represents dosage of a drug for an illness and the vertical axis
represents a severity of illness measure. As dosage rises, severity of illness goes down. But at
some point, the patient begins to experience negative side effects associated with too high a
dosage, and the severity of illness begins to increase again.

Variables
You won't be able to do very much in research unless you know how to talk about variables. A
variable is any entity that can take on different values. OK, so what does that mean? Anything
that can vary can be considered a variable. For instance, age can be considered a variable
because age can take different values for different people or for the same person at different
times. Similarly, country can be considered a variable because a person's country can be assigned
a value.
Variables aren't always 'quantitative' or numerical. The variable 'gender' consists of two text
values: 'male' and 'female'. We can, if it is useful, assign quantitative values instead of (or in
place of) the text values, but we don't have to assign numbers in order for something to be a
variable. It's also important to realize that variables aren't only things that we measure in the
traditional sense. For instance, in much social research and in program evaluation, we consider
the treatment or program to be made up of one or more variables (i.e., the 'cause' can be
considered a variable). An educational program can have varying amounts of 'time on task',
'classroom settings', 'student-teacher ratios', and so on. So even the program can be considered a
variable (which can be made up of a number of sub-variables).
An attribute is a specific value on a variable. For instance, the variable sex or gender has two
attributes: male and female. Or, the variable agreement might be defined as having five
attributes:
 • 1 = strongly disagree
• 2 = disagree
• 3 = neutral
• 4 = agree
• 5 = strongly agree
Another important distinction having to do with the term 'variable' is the distinction between an
independent and dependent variable. This distinction is particularly relevant when you are
investigating cause-effect relationships. It took me the longest time to learn this distinction. (Of
course, I'm someone who gets confused about the signs for 'arrivals' and 'departures' at airports --
do I go to arrivals because I'm arriving at the airport or does the person I'm picking up go to
arrivals because they're arriving on the plane!). I originally thought that an independent variable
was one that would be free to vary or respond to some program or treatment, and that a
dependent variable must be one that depends on my efforts (that is, it's the treatment). But this is
entirely backwards! In fact the independent variable is what you (or nature) manipulates -- a
treatment or program or cause. The dependent variable is what is affected by the independent
variable -- your effects or outcomes. For example, if you are studying the effects of a new
educational program on student achievement, the program is the independent variable and your
measures of achievement are the dependent ones.
Finally, there are two traits of variables that should always be achieved. Each variable should be
exhaustive, it should include all possible answerable responses. For instance, if the variable is
"religion" and the only options are "Protestant", "Jewish", and "Muslim", there are quite a few
religions I can think of that haven't been included. The list does not exhaust all possibilities. On
the other hand, if you exhaust all the possibilities with some variables -- religion being one of
them -- you would simply have too many responses. The way to deal with this is to explicitly list
the most common attributes and then use a general category like "Other" to account for all
remaining ones. In addition to being exhaustive, the attributes of a variable should be mutually
exclusive, no respondent should be able to have two attributes simultaneously. While this might
seem obvious, it is often rather tricky in practice. For instance, you might be tempted to represent
the variable "Employment Status" with the two attributes "employed" and "unemployed." But
these attributes are not necessarily mutually exclusive -- a person who is looking for a second job
while employed would be able to check both attributes! But don't we often use questions on
surveys that ask the respondent to "check all that apply" and then list a series of categories? Yes,
we do, but technically speaking, each of the categories in a question like that is its own variable
and is treated dichotomously as either "checked" or "unchecked", attributes that are mutually
exclusive.

Hypotheses
An hypothesis is a specific statement of prediction. It describes in concrete (rather than
theoretical) terms what you expect will happen in your study. Not all studies have hypotheses.
Sometimes a study is designed to be exploratory (see inductive research). There is no formal
hypothesis, and perhaps the purpose of the study is to explore some area more thoroughly in
order to develop some specific hypothesis or prediction that can be tested in future research. A
single study may have one or many hypotheses.
Actually, whenever I talk about an hypothesis, I am really thinking simultaneously about two
hypotheses. Let's say that you predict that there will be a relationship between two variables in
your study. The way we would formally set up the hypothesis test is to formulate two hypothesis
statements, one that describes your prediction and one that describes all the other possible
outcomes with respect to the hypothesized relationship. Your prediction is that variable A and
variable B will be related (you don't care whether it's a positive or negative relationship). Then
the only other possible outcome would be that variable A and variable B are not related. Usually,
we call the hypothesis that you support (your prediction) the alternative hypothesis, and we call
the hypothesis that describes the remaining possible outcomes the null hypothesis. Sometimes
we use a notation like HA or H1 to represent the alternative hypothesis or your prediction, and HO
or H0 to represent the null case. You have to be careful here, though. In some studies, your
prediction might very well be that there will be no difference or change. In this case, you are
essentially trying to find support for the null hypothesis and you are opposed to the alternative.
If your prediction specifies a direction, and the null therefore is the no difference prediction and
the prediction of the opposite direction, we call this a one-tailed hypothesis. For instance, let's
imagine that you are investigating the effects of a new employee training program and that you
believe one of the outcomes will be that there will be less employee absenteeism. Your two
hypotheses might be stated something like this:
The null hypothesis for this study is:
HO: As a result of the XYZ company employee training program, there will either be no
significant difference in employee absenteeism or there will be a significant increase.
which is tested against the alternative hypothesis:
HA: As a result of the XYZ company employee training program, there will be a significant
decrease in employee absenteeism.
In the figure on the left, we see this situation
illustrated graphically. The alternative hypothesis
-- your prediction that the program will decrease
absenteeism -- is shown there. The null must
account for the other two possible conditions: no
difference, or an increase in absenteeism. The
figure shows a hypothetical distribution of
absenteeism differences. We can see that the term
"one-tailed" refers to the tail of the distribution on
the outcome variable.
When your prediction does not specify a direction, we say you have a two-tailed hypothesis. For
instance, let's assume you are studying a new drug treatment for depression. The drug has gone
through some initial animal trials, but has not yet been tested on humans. You believe (based on
theory and the previous research) that the drug will have an effect, but you are not confident
enough to hypothesize a direction and say the drug will reduce depression (after all, you've seen
more than enough promising drug treatments come along that eventually were shown to have
severe side effects that actually worsened symptoms). In this case, you might state the two
hypotheses like this:
The null hypothesis for this study is:
HO: As a result of 300mg./day of the ABC drug, there will be no significant difference in
depression.
which is tested against the alternative hypothesis:
HA: As a result of 300mg./day of the ABC drug, there will be a significant difference in
depression.
The figure on the right illustrates this two-tailed
prediction for this case. Again, notice that the
term "two-tailed" refers to the tails of the
distribution for your outcome variable.
The important thing to remember about stating
hypotheses is that you formulate your prediction
(directional or not), and then you formulate a
second hypothesis that is mutually exclusive of
the first and incorporates all possible alternative
outcomes for that case. When your study analysis
is completed, the idea is that you will have to choose between the two hypotheses. If your
prediction was correct, then you would (usually) reject the null hypothesis and accept the
alternative. If your original prediction was not supported in the data, then you will accept the null
hypothesis and reject the alternative. The logic of hypothesis testing is based on these two basic
principles:
• the formulation of two mutually exclusive hypothesis statements that, together, exhaust
all possible outcomes
• the testing of these so that one is necessarily accepted and the other rejected
OK, I know it's a convoluted, awkward and formalistic way to ask research questions. But it
encompasses a long tradition in statistics called the hypothetical-deductive model, and
sometimes we just have to do things because they're traditions. And anyway, if all of this
hypothesis testing was easy enough so anybody could understand it, how do you think
statisticians would stay employed?

Types of Data
We'll talk about data in lots of places in The Knowledge Base, but here I just want to make a
fundamental distinction between two types of data: qualitative and quantitative. The way we
typically define them, we call data 'quantitative' if it is in numerical form and 'qualitative' if it is
not. Notice that qualitative data could be much more than just words or text. Photographs,
videos, sound recordings and so on, can be considered qualitative data.
Personally, while I find the distinction between qualitative and quantitative data to have some
utility, I think most people draw too hard a distinction, and that can lead to all sorts of confusion.
In some areas of social research, the qualitative-quantitative distinction has led to protracted
arguments with the proponents of each arguing the superiority of their kind of data over the
other. The quantitative types argue that their data is 'hard', 'rigorous', 'credible', and 'scientific'.
The qualitative proponents counter that their data is 'sensitive', 'nuanced', 'detailed', and
'contextual'.
For many of us in social research, this kind of polarized debate has become less than productive.
And, it obscures the fact that qualitative and quantitative data are intimately related to each other.
All quantitative data is based upon qualitative judgments; and all qualitative data can be
described and manipulated numerically. For instance, think about a very common quantitative
measure in social research -- a self esteem scale. The researchers who develop such instruments
had to make countless judgments in constructing them: how to define self esteem; how to
distinguish it from other related concepts; how to word potential scale items; how to make sure
the items would be understandable to the intended respondents; what kinds of contexts it could
be used in; what kinds of cultural and language constraints might be present; and on and on. The
researcher who decides to use such a scale in their study has to make another set of judgments:
how well does the scale measure the intended concept; how reliable or consistent is it; how
appropriate is it for the research context and intended respondents; and on and on. Believe it or
not, even the respondents make many judgments when filling out such a scale: what is meant by
various terms and phrases; why is the researcher giving this scale to them; how much energy and
effort do they want to expend to complete it, and so on. Even the consumers and readers of the
research will make lots of judgments about the self esteem measure and its appropriateness in
that research context. What may look like a simple, straightforward, cut-and-dried quantitative
measure is actually based on lots of qualitative judgments made by lots of different people.
On the other hand, all qualitative
information can be easily converted into
quantitative, and there are many times
when doing so would add considerable
value to your research. The simplest
way to do this is to divide the qualitative
information into units and number them!
I know that sounds trivial, but even that
simple nominal enumeration can enable
you to organize and process qualitative
information more efficiently. Perhaps
more to the point, we might take text
information (say, excerpts from
transcripts) and pile these excerpts into
piles of similar statements. When we do
something even as easy as this simple grouping or piling task, we can describe the results
quantitatively. For instance, if we had ten statements and we grouped these into five piles (as
shown in the figure), we could describe the piles using a 10 x 10 table of 0's and 1's. If two
statements were placed together in the same pile, we would put a 1 in their row-column juncture.
If two statements were placed in different piles, we would use a 0. The resulting matrix or table
describes the grouping of the ten statements in terms of their similarity. Even though the data in
this example consists of qualitative statements (one per card), the result of our simple qualitative
procedure (grouping similar excerpts into the same piles) is quantitative in nature. "So what?"
you ask. Once we have the data in numerical form, we can manipulate it numerically. For
instance, we could have five different judges sort the 10 excerpts and obtain a 0-1 matrix like this
for each judge. Then we could average the five matrices into a single one that shows the
proportions of judges who grouped each pair together. This proportion could be considered an
estimate of the similarity (across independent judges) of the excerpts. While this might not seem
too exciting or useful, it is exactly this kind of procedure that I use as an integral part of the
process of developing 'concept maps' of ideas for groups of people (something that is useful!).

Unit of Analysis
One of the most important ideas in a research project is the unit of analysis. The unit of analysis
is the major entity that you are analyzing in your study. For instance, any of the following could
be a unit of analysis in a study:
 • individuals
• groups
• artifacts (books, photos, newspapers)
• geographical units (town, census tract, state)
• social interactions (dyadic relations, divorces, arrests)
Why is it called the 'unit of analysis' and not something else (like, the unit of sampling)? Because
it is the analysis you do in your study that determines what the unit is. For instance, if you are
comparing the children in two classrooms on achievement test scores, the unit is the individual
child because you have a score for each child. On the other hand, if you are comparing the two
classes on classroom climate, your unit of analysis is the group, in this case the classroom,
because you only have a classroom climate score for the class as a whole and not for each
individual student. For different analyses in the same study you may have different units of
analysis. If you decide to base an analysis on student scores, the individual is the unit. But you
might decide to compare average classroom performance. In this case, since the data that goes
into the analysis is the average itself (and not the individuals' scores) the unit of analysis is
actually the group. Even though you had data at the student level, you use aggregates in the
analysis. In many areas of social research these hierarchies of analysis units have become
particularly important and have spawned a whole area of statistical analysis sometimes referred
to as hierarchical modeling. This is true in education, for instance, where we often compare
classroom performance but collected achievement data at the individual student level.

Two Research Fallacies

A fallacy is an error in reasoning, usually based on mistaken assumptions. Researchers are very
familiar with all the ways they could go wrong, with the fallacies they are susceptible to. Here, I
discuss two of the most important.
The ecological fallacy occurs when you make conclusions about individuals based only on
analyses of group data. For instance, assume that you measured the math scores of a particular
classroom and found that they had the highest average score in the district. Later (probably at the
mall) you run into one of the kids from that class and you think to yourself "she must be a math
whiz." Aha! Fallacy! Just because she comes from the class with the highest average doesn't
mean that she is automatically a high-scorer in math. She could be the lowest math scorer in a
class that otherwise consists of math geniuses!
An exception fallacy is sort of the reverse of the ecological fallacy. It occurs when you reach a
group conclusion on the basis of exceptional cases. This is the kind of fallacious reasoning that is
at the core of a lot of sexism and racism. The stereotype is of the guy who sees a woman make a
driving error and concludes that "women are terrible drivers." Wrong! Fallacy!
Both of these fallacies point to some of the traps that exist in both research and everyday
reasoning. They also point out how important it is that we do research. We need to determine
empirically how individuals perform (not just rely on group averages). Similarly, we need to
look at whether there are correlations between certain behaviors and certain groups (you might
look at the whole controversy around the book The Bell Curve as an attempt to examine whether
the supposed relationship between race and IQ is real or a fallacy.

Philosophy of Research
You probably think of research as something very abstract and complicated. It can be, but you'll
see (I hope) that if you understand the different parts or phases of a research project and how
these fit together, it's not nearly as complicated as it may seem at first glance. A research project
has a well-known structure -- a beginning, middle and end. We introduce the basic phases of a
research project in The Structure of Research. In that section, we also introduce some important
distinctions in research: the different types of questions you can ask in a research project; and,
the major components or parts of a research project.
Before the modern idea of research emerged, we had a term for what philosophers used to call
research -- logical reasoning. So, it should come as no surprise that some of the basic distinctions
in logic have carried over into contemporary research. In Systems of Logic we discuss how two
major logical systems, the inductive and deductive methods of reasoning, are related to modern
research.
OK, you knew that no introduction would be complete without considering something having to
do with assumptions and philosophy. (I thought I very cleverly snuck in the stuff about logic in
the last paragraph). All research is based on assumptions about how the world is perceived and
how we can best come to understand it. Of course, nobody really knows how we can best
understand the world, and philosophers have been arguing about that very question for at least
two millennia now, so all we're going to do is look at how most contemporary social scientists
approach the question of how we know about the world around us. We consider two major
philosophical schools of thought -- Positivism and Post-Positivism -- that are especially
important perspectives for contemporary social research (OK, I'm only considering positivism
and post-positivism here because these are the major schools of thought. Forgive me for not
considering the hotly debated alternatives like relativism, subjectivism, hermeneutics,
deconstructivism, constructivism, feminism, etc. If you really want to cover that stuff, start your
own Web site and send me your URL to stick in here).
Quality is one of the most important issues in research. We introduce the idea of validity to refer
to the quality of various conclusions you might reach based on a research project. Here's where
I've got to give you the pitch about validity. When I mention validity, most students roll their
eyes, curl up into a fetal position or go to sleep. They think validity is just something abstract and
philosophical (and I guess it is at some level). But I think if you can understand validity -- the
principles that we use to judge the quality of research -- you'll be able to do much more than just
complete a research project. You'll be able to be a virtuoso at research, because you'll have an
understanding of why we need to do certain things in order to assure quality. You won't just be
plugging in standard procedures you learned in school -- sampling method X, measurement tool
Y -- you'll be able to help create the next generation of research technology. Enough for now --
more on this later.

Structure of Research
Most research projects share the same general structure. You might think of this structure as
following the shape of an hourglass. The research process usually starts with a broad area of
interest, the initial problem that the researcher wishes to study. For instance, the researcher could
be interested in how to use computers to improve the performance of students in mathematics.
But this initial interest is far too broad to study in any single research project (it might not even
be addressable in a lifetime of research). The researcher has to narrow the question down to one
that can reasonably be studied in a research project. This might involve formulating a hypothesis
or a focus question. For instance, the researcher might hypothesize that a particular method of
computer instruction in math will improve the ability of elementary school students in a specific
district. At the narrowest point of the research hourglass, the researcher is engaged in direct
measurement or observation of the question of interest.

Once the basic data is collected, the researcher begins to try to understand it, usually by
analyzing it in a variety of ways. Even for a single hypothesis there are a number of analyses a
researcher might typically conduct. At this point, the researcher begins to formulate some initial
conclusions about what happened as a result of the computerized math program. Finally, the
researcher often will attempt to address the original broad question of interest by generalizing
from the results of this specific study to other related situations. For instance, on the basis of
strong results indicating that the math program had a positive effect on student performance, the
researcher might conclude that other school districts similar to the one in the study might expect
similar results.
Components of a Study
What are the basic components or parts of a research study? Here, we'll describe the basic
components involved in a causal study. Because causal studies presuppose descriptive and
relational questions, many of the components of causal studies will also be found in those others.
Most social research originates from some general problem or question. You might, for instance,
be interested in what programs enable the unemployed to get jobs. Usually, the problem is broad
enough that you could not hope to address it adequately in a single research study. Consequently,
we typically narrow the problem down to a more specific research question that we can hope to
address. The research question is often stated in the context of some theory that has been
advanced to address the problem. For instance, we might have the theory that ongoing support
services are needed to assure that the newly employed remain employed. The research question
is the central issue being addressed in the study and is often phrased in the language of theory.
For instance, a research question might be:
Is a program of supported employment more effective (than no program at all) at keeping newly
employed persons on the job?
The problem with such a question is that it is still too general to be studied directly.
Consequently, in most research we develop an even more specific statement, called an
hypothesis that describes in operational terms exactly what we think will happen in the study.
For instance, the hypothesis for our employment study might be something like:
The Metropolitan Supported Employment Program will significantly increase rates of
employment after six months for persons who are newly employed (after being out of work for at
least one year) compared with persons who receive no comparable program.
Notice that this hypothesis is specific enough that a reader can understand quite well what the
study is trying to assess.
In causal studies, we have at least two major variables of interest, the cause and the effect.
Usually the cause is some type of event, program, or treatment. We make a distinction between
causes that the researcher can control (such as a program) versus causes that occur naturally or
outside the researcher's influence (such as a change in interest rates, or the occurrence of an
earthquake). The effect is the outcome that you wish to study. For both the cause and effect we
make a distinction between our idea of them (the construct) and how they are actually manifested
in reality. For instance, when we think about what a program of support services for the newly
employed might be, we are thinking of the "construct." On the other hand, the real world is not
always what we think it is. In research, we remind ourselves of this by distinguishing our view of
an entity (the construct) from the entity as it exists (the operationalization). Ideally, we would
like the two to agree.
Social research is always conducted in a social context. We ask people questions, or observe
families interacting, or measure the opinions of people in a city. An important component of a
research project is the units that participate in the project. Units are directly related to the
question of sampling. In most projects we cannot involve all of the people we might like to
involve. For instance, in studying a program of support services for the newly employed we can't
possibly include in our study everyone in the world, or even in the country, who is newly
employed. Instead, we have to try to obtain a representative sample of such people. When
sampling, we make a distinction between the theoretical population of interest to our study and
the final sample that we actually measure in our study. Usually the term "units" refers to the
people that we sample and from whom we gather information. But for some projects the units are
organizations, groups, or geographical entities like cities or towns. Sometimes our sampling
strategy is multi-level: we sample a number of cities and within them sample families.
In causal studies, we are interested in the effects of some cause on one or more outcomes. The
outcomes are directly related to the research problem -- we are usually most interested in
outcomes that are most reflective of the problem. In our hypothetical supported employment
study, we would probably be most interested in measures of employment -- is the person
currently employed, or, what is their rate of absenteeism.
Finally, in a causal study we usually are comparing the effects of our cause of interest (e.g., the
program) relative to other conditions (e.g., another program or no program at all). Thus, a key
component in a causal study concerns how we decide what units (e.g., people) receive our
program and which are placed in an alternative condition. This issue is directly related to the
research design that we use in the study. One of the central questions in research design is
determining how people wind up in or are placed in various programs or treatments that we are
comparing.
These, then, are the major components in a causal study:
• The Research Problem
• The Research Question
• The Program (Cause)
• The Units
• The Outcomes (Effect)
• The Design

• Deduction & Induction

• Deductive and Inductive Thinking
•
• In logic, we often refer to the two broad methods of reasoning as the deductive and
inductive approaches.
• Deductive reasoning works
from the more general to
the more specific.
Sometimes this is
informally called a "top-
down" approach. We
might begin with thinking
up a theory about our topic
of interest. We then narrow
that down into more
specific hypotheses that we can test. We narrow down even further when we collect
observations to address the hypotheses. This ultimately leads us to be able to test the
hypotheses with specific data -- a confirmation (or not) of our original theories.
• Inductive reasoning works
the other way, moving
from specific observations
to broader generalizations
and theories. Informally,
we sometimes call this a
"bottom up" approach
(please note that it's
"bottom up" and not
"bottoms up" which is the
kind of thing the bartender
says to customers when he's trying to close for the night!). In inductive reasoning, we
begin with specific observations and measures, begin to detect patterns and regularities,
formulate some tentative hypotheses that we can explore, and finally end up developing
some general conclusions or theories.
• These two methods of reasoning have a very different "feel" to them when you're
conducting research. Inductive reasoning, by its very nature, is more open-ended and
exploratory, especially at the beginning. Deductive reasoning is more narrow in nature
and is concerned with testing or confirming hypotheses. Even though a particular study
may look like it's purely deductive (e.g., an experiment designed to test the hypothesized
effects of some treatment on some outcome), most social research involves both
inductive and deductive reasoning processes at some time in the project. In fact, it doesn't
take a rocket scientist to see that we could assemble the two graphs above into a single
 circular one that continually cycles from theories down to observations and back up again
to theories. Even in the most constrained experiment, the researchers may observe
patterns in the data that lead them to develop new theories.

Positivism & Post-Positivism

Let's start our very brief discussion of philosophy of science with a simple distinction between
epistemology and methodology. The term epistemology comes from the Greek word epistêmê,
their term for knowledge. In simple terms, epistemology is the philosophy of knowledge or of
how we come to know. Methodology is also concerned with how we come to know, but is much
more practical in nature. Methodology is focused on the specific ways -- the methods -- that we
can use to try to understand our world better. Epistemology and methodology are intimately
related: the former involves the philosophy of how we come to know the world and the latter
involves the practice.
When most people in our society think about science, they think about some guy in a white lab
coat working at a lab bench mixing up chemicals. They think of science as boring, cut-and-dry,
and they think of the scientist as narrow-minded and esoteric (the ultimate nerd -- think of the
humorous but nonetheless mad scientist in the Back to the Future movies, for instance). A lot of
our stereotypes about science come from a period where science was dominated by a particular
philosophy -- positivism -- that tended to support some of these views. Here, I want to suggest
(no matter what the movie industry may think) that science has moved on in its thinking into an
era of post-positivism where many of those stereotypes of the scientist no longer hold up.
Let's begin by considering what positivism is. In its broadest sense, positivism is a rejection of
metaphysics (I leave it you to look up that term if you're not familiar with it). It is a position that
holds that the goal of knowledge is simply to describe the phenomena that we experience. The
purpose of science is simply to stick to what we can observe and measure. Knowledge of
anything beyond that, a positivist would hold, is impossible. When I think of positivism (and the
related philosophy of logical positivism) I think of the behaviorists in mid-20th Century
psychology. These were the mythical 'rat runners' who believed that psychology could only study
what could be directly observed and measured. Since we can't directly observe emotions,
thoughts, etc. (although we may be able to measure some of the physical and physiological
accompaniments), these were not legitimate topics for a scientific psychology. B.F. Skinner
argued that psychology needed to concentrate only on the positive and negative reinforcers of
behavior in order to predict how people will behave -- everything else in between (like what the
person is thinking) is irrelevant because it can't be measured.
In a positivist view of the world, science was seen as the way to get at truth, to understand the
world well enough so that we might predict and control it. The world and the universe were
deterministic -- they operated by laws of cause and effect that we could discern if we applied the
unique approach of the scientific method. Science was largely a mechanistic or mechanical
affair. We use deductive reasoning to postulate theories that we can test. Based on the results of
our studies, we may learn that our theory doesn't fit the facts well and so we need to revise our
theory to better predict reality. The positivist believed in empiricism -- the idea that observation
and measurement was the core of the scientific endeavor. The key approach of the scientific
method is the experiment, the attempt to discern natural laws through direct manipulation and
observation.
OK, I am exaggerating the positivist position (although you may be amazed at how close to this
some of them actually came) in order to make a point. Things have changed in our views of
science since the middle part of the 20th century. Probably the most important has been our shift
away from positivism into what we term post-positivism. By post-positivism, I don't mean a
slight adjustment to or revision of the positivist position -- post-positivism is a wholesale
rejection of the central tenets of positivism. A post-positivist might begin by recognizing that the
way scientists think and work and the way we think in our everyday life are not distinctly
different. Scientific reasoning and common sense reasoning are essentially the same process.
There is no difference in kind between the two, only a difference in degree. Scientists, for
example, follow specific procedures to assure that observations are verifiable, accurate and
consistent. In everyday reasoning, we don't always proceed so carefully (although, if you think
about it, when the stakes are high, even in everyday life we become much more cautious about
measurement. Think of the way most responsible parents keep continuous watch over their
infants, noticing details that non-parents would never detect).
One of the most common forms of post-positivism is a philosophy called critical realism. A
critical realist believes that there is a reality independent of our thinking about it that science can
study. (This is in contrast with a subjectivist who would hold that there is no external reality --
we're each making this all up!). Positivists were also realists. The difference is that the post-
positivist critical realist recognizes that all observation is fallible and has error and that all theory
is revisable. In other words, the critical realist is critical of our ability to know reality with
certainty. Where the positivist believed that the goal of science was to uncover the truth, the
post-positivist critical realist believes that the goal of science is to hold steadfastly to the goal of
getting it right about reality, even though we can never achieve that goal! Because all
measurement is fallible, the post-positivist emphasizes the importance of multiple measures and
observations, each of which may possess different types of error, and the need to use
triangulation across these multiple errorful sources to try to get a better bead on what's
happening in reality. The post-positivist also believes that all observations are theory-laden and
that scientists (and everyone else, for that matter) are inherently biased by their cultural
experiences, world views, and so on. This is not cause to give up in despair, however. Just
because I have my world view based on my experiences and you have yours doesn't mean that
we can't hope to translate from each other's experiences or understand each other. That is, post-
positivism rejects the relativist idea of the incommensurability of different perspectives, the idea
that we can never understand each other because we come from different experiences and
cultures. Most post-positivists are constructivists who believe that we each construct our view of
the world based on our perceptions of it. Because perception and observation is fallible, our
constructions must be imperfect. So what is meant by objectivity in a post-positivist world?
Positivists believed that objectivity was a characteristic that resided in the individual scientist.
Scientists are responsible for putting aside their biases and beliefs and seeing the world as it
'really' is. Post-positivists reject the idea that any individual can see the world perfectly as it
really is. We are all biased and all of our observations are affected (theory-laden). Our best hope
for achieving objectivity is to triangulate across multiple fallible perspectives! Thus, objectivity
is not the characteristic of an individual, it is inherently a social phenomenon. It is what multiple
individuals are trying to achieve when they criticize each other's work. We never achieve
objectivity perfectly, but we can approach it. The best way for us to improve the objectivity of
what we do is to do it within the context of a broader contentious community of truth-seekers
(including other scientists) who criticize each other's work. The theories that survive such intense
scrutiny are a bit like the species that survive in the evolutionary struggle. (This is sometimes
called the natural selection theory of knowledge and holds that ideas have 'survival value' and
that knowledge evolves through a process of variation, selection and retention). They have
adaptive value and are probably as close as our species can come to being objective and
understanding reality.
Clearly, all of this stuff is not for the faint-of-heart. I've seen many a graduate student get lost in
the maze of philosophical assumptions that contemporary philosophers of science argue about.
And don't think that I believe this is not important stuff. But, in the end, I tend to turn pragmatist
on these matters. Philosophers have been debating these issues for thousands of years and there
is every reason to believe that they will continue to debate them for thousands of years more.
Those of us who are practicing scientists should check in on this debate from time to time
(perhaps every hundred years or so would be about right). We should think about the
assumptions we make about the world when we conduct research. But in the meantime, we can't
wait for the philosophers to settle the matter. After all, we do have our own work to do!

Introduction to Validity
Validity:
the best available approximation to the truth of a given
proposition, inference, or conclusion
The first thing we have to ask is: "validity of what?" When we think about validity in research,
most of us think about research components. We might say that a measure is a valid one, or that
a valid sample was drawn, or that the design had strong validity. But all of those statements are
technically incorrect. Measures, samples and designs don't 'have' validity -- only propositions can
be said to be valid. Technically, we should say that a measure leads to valid conclusions or that a
sample enables valid inferences, and so on. It is a proposition, inference or conclusion that can
'have' validity.
We make lots of different inferences or conclusions while conducting research. Many of these
are related to the process of doing research and are not the major hypotheses of the study.
Nevertheless, like the bricks that go into building a wall, these intermediate process and
methodological propositions provide the foundation for the substantive conclusions that we wish
to address. For instance, virtually all social research involves measurement or observation. And,
whenever we measure or observe we are concerned with whether we are measuring what we
intend to measure or with how our observations are influenced by the circumstances in which
they are made. We reach conclusions about the quality of our measures -- conclusions that will
play an important role in addressing the broader substantive issues of our study. When we talk
about the validity of research, we are often referring to these to the many conclusions we reach
about the quality of different parts of our research methodology.
We subdivide validity into four types. Each type addresses a specific methodological question. In
order to understand the types of validity, you have to know something about how we investigate
a research question. Because all four validity types are really only operative when studying
causal questions, we will use a causal study to set the context.
The figure shows that there are really two realms that are involved in research. The first, on the
top, is the land of theory. It is what goes on inside our heads as researchers. It is were we keep
our theories about how the world operates. The second, on the bottom, is the land of
observations. It is the real world into which we translate our ideas -- our programs, treatments,
measures and observations. When we conduct research, we are continually flitting back and forth
between these two realms, between what we think about the world and what is going on in it.
When we are investigating a cause-effect relationship, we have a theory (implicit or otherwise)
of what the cause is (the cause construct). For instance, if we are testing a new educational
program, we have an idea of what it would look like ideally. Similarly, on the effect side, we
have an idea of what we are ideally trying to affect and measure (the effect construct). But each
of these, the cause and the effect, has to be translated into real things, into a program or treatment
and a measure or observational method. We use the term operationalization to describe the act
of translating a construct into its manifestation. In effect, we take our idea and describe it as a
series of operations or procedures. Now, instead of it only being an idea in our minds, it becomes
a public entity that anyone can look at and examine for themselves. It is one thing, for instance,
for you to say that you would like to measure self-esteem (a construct). But when you show a
ten-item paper-and-pencil self-esteem measure that you developed for that purpose, others can
look at it and understand more clearly what you intend by the term self-esteem.
Now, back to explaining the four validity types. They build on one another, with two of them
(conclusion and internal) referring to the land of observation on the bottom of the figure, one of
them (construct) emphasizing the linkages between the bottom and the top, and the last (external)
being primarily concerned about the range of our theory on the top. Imagine that we wish to
examine whether use of a World Wide Web (WWW) Virtual Classroom improves student
understanding of course material. Assume that we took these two constructs, the cause construct
(the WWW site) and the effect (understanding), and operationalized them -- turned them into
realities by constructing the WWW site and a measure of knowledge of the course material. Here
are the four validity types and the question each addresses:
Conclusion Validity: In this study, is there a relationship between the two variables?
In the context of the example we're considering, the question might be worded: in this study, is
there a relationship between the WWW site and knowledge of course material? There are several
conclusions or inferences we might draw to answer such a question. We could, for example,
conclude that there is a relationship. We might conclude that there is a positive relationship. We
might infer that there is no relationship. We can assess the conclusion validity of each of these
conclusions or inferences.
Internal Validity: Assuming that there is a relationship in this study, is the relationship
a causal one?
Just because we find that use of the WWW site and knowledge are correlated, we can't
necessarily assume that WWW site use causes the knowledge. Both could, for example, be
caused by the same factor. For instance, it may be that wealthier students who have greater
resources would be more likely to use have access to a WWW site and would excel on objective
tests. When we want to make a claim that our program or treatment caused the outcomes in our
study, we can consider the internal validity of our causal claim.
Construct Validity: Assuming that there is a causal relationship in this study, can we
claim that the program reflected well our construct of the program and that our
measure reflected well our idea of the construct of the measure?
In simpler terms, did we implement the program we intended to implement and did we measure
the outcome we wanted to measure? In yet other terms, did we operationalize well the ideas of
the cause and the effect? When our research is over, we would like to be able to conclude that we
did a credible job of operationalizing our constructs -- we can assess the construct validity of this
conclusion.
External Validity: Assuming that there is a causal relationship in this study between
the constructs of the cause and the effect, can we generalize this effect to other
persons, places or times?
We are likely to make some claims that our research findings have implications for other groups
and individuals in other settings and at other times. When we do, we can examine the external
validity of these
claims.
Notice how the
question that each
validity type
addresses
presupposes an
affirmative answer
to the previous
one. This is what
we mean when we
say that the
validity types build
on one another.
The figure shows the idea of cumulativeness as a staircase, along with the key question for each
validity type.
For any inference or conclusion, there are always possible threats to validity -- reasons the
conclusion or inference might be wrong. Ideally, one tries to reduce the plausibility of the most
likely threats to validity, thereby leaving as most plausible the conclusion reached in the study.
For instance, imagine a study examining whether there is a relationship between the amount of
training in a specific technology and subsequent rates of use of that technology. Because the
interest is in a relationship, it is considered an issue of conclusion validity. Assume that the study
is completed and no significant correlation between amount of training and adoption rates is
found. On this basis it is concluded that there is no relationship between the two. How could this
conclusion be wrong -- that is, what are the "threats to validity"? For one, it's possible that there
isn't sufficient statistical power to detect a relationship even if it exists. Perhaps the sample size
is too small or the measure of amount of training is unreliable. Or maybe assumptions of the
correlational test are violated given the variables used. Perhaps there were random irrelevancies
in the study setting or random heterogeneity in the respondents that increased the variability in
the data and made it harder to see the relationship of interest. The inference that there is no
relationship will be stronger -- have greater conclusion validity -- if one can show that these
alternative explanations are not credible. The distributions might be examined to see if they
conform with assumptions of the statistical test, or analyses conducted to determine whether
there is sufficient statistical power.
The theory of validity, and the many lists of specific threats, provide a useful scheme for
assessing the quality of research conclusions. The theory is general in scope and applicability,
well-articulated in its philosophical suppositions, and virtually impossible to explain adequately
in a few minutes. As a framework for judging the quality of evaluations it is indispensable and
well worth understanding.

Ethics in Research
We are going through a time of profound change in our understanding of the ethics of applied
social research. From the time immediately after World War II until the early 1990s, there was a
gradually developing consensus about the key ethical principles that should underlie the research
endeavor. Two marker events stand out (among many others) as symbolic of this consensus. The
Nuremberg War Crimes Trial following World War II brought to public view the ways German
scientists had used captive human subjects as subjects in oftentimes gruesome experiments. In
the 1950s and 1960s, the Tuskegee Syphilis Study involved the withholding of known effective
treatment for syphilis from African-American participants who were infected. Events like these
forced the reexamination of ethical standards and the gradual development of a consensus that
potential human subjects needed to be protected from being used as 'guinea pigs' in scientific
research.
By the 1990s, the dynamics of the situation changed. Cancer patients and persons with AIDS
fought publicly with the medical research establishment about the long time needed to get
approval for and complete research into potential cures for fatal diseases. In many cases, it is the
ethical assumptions of the previous thirty years that drive this 'go-slow' mentality. After all, we
would rather risk denying treatment for a while until we achieve enough confidence in a
treatment, rather than run the risk of harming innocent people (as in the Nuremberg and
Tuskegee events). But now, those who were threatened with fatal illness were saying to the
research establishment that they wanted to be test subjects, even under experimental conditions
of considerable risk. You had several very vocal and articulate patient groups who wanted to be
experimented on coming up against an ethical review system that was designed to protect them
from being experimented on.
Although the last few years in the ethics of research have been tumultuous ones, it is beginning
to appear that a new consensus is evolving that involves the stakeholder groups most affected by
a problem participating more actively in the formulation of guidelines for research. While it's not
entirely clear, at present, what the new consensus will be, it is almost certain that it will not fall
at either extreme: protecting against human experimentation at all costs vs. allowing anyone who
is willing to be experimented on.
Ethical Issues
There are a number of key phrases that describe the system of ethical protections that the
contemporary social and medical research establishment have created to try to protect better the
rights of their research participants. The principle of voluntary participation requires that people
not be coerced into participating in research. This is especially relevant where researchers had
previously relied on 'captive audiences' for their subjects -- prisons, universities, and places like
that. Closely related to the notion of voluntary participation is the requirement of informed
consent. Essentially, this means that prospective research participants must be fully informed
about the procedures and risks involved in research and must give their consent to participate.
Ethical standards also require that researchers not put participants in a situation where they might
be at risk of harm as a result of their participation. Harm can be defined as both physical and
psychological. There are two standards that are applied in order to help protect the privacy of
research participants. Almost all research guarantees the participants confidentiality -- they are
assured that identifying information will not be made available to anyone who is not directly
involved in the study. The stricter standard is the principle of anonymity which essentially means
that the participant will remain anonymous throughout the study -- even to the researchers
themselves. Clearly, the anonymity standard is a stronger guarantee of privacy, but it is
sometimes difficult to accomplish, especially in situations where participants have to be
measured at multiple time points (e.g., a pre-post study). Increasingly, researchers have had to
deal with the ethical issue of a person's right to service. Good research practice often requires the
use of a no-treatment control group -- a group of participants who do not get the treatment or
program that is being studied. But when that treatment or program may have beneficial effects,
persons assigned to the no-treatment control may feel their rights to equal access to services are
being curtailed.
Even when clear ethical standards and principles exist, there will be times when the need to do
accurate research runs up against the rights of potential participants. No set of standards can
possibly anticipate every ethical circumstance. Furthermore, there needs to be a procedure that
assures that researchers will consider all relevant ethical issues in formulating research plans. To
address such needs most institutions and organizations have formulated an Institutional Review
Board (IRB), a panel of persons who reviews grant proposals with respect to ethical implications
and decides whether additional actions need to be taken to assure the safety and rights of
participants. By reviewing proposals for research, IRBs also help to protect both the organization
and the researcher against potential legal implications of neglecting to address important ethical
issues of participants.

Conceptualizing
One of the most difficult aspects of research -- and one of the least discussed -- is how to develop
the idea for the research project in the first place. In training students, most faculty just assume
that if you read enough of the research in an area of interest, you will somehow magically be
able to produce sensible ideas for further research. Now, that may be true. And heaven knows
that's the way we've been doing this higher education thing for some time now. But it troubles
me that we haven't been able to do a better job of helping our students learn how to formulate
good research problems. One thing we can do (and some texts at least cover this at a surface
level) is to give students a better idea of how professional researchers typically generate research
ideas. Some of this is introduced in the discussion of problem formulation in applied social
research.
But maybe we can do even better than that. Why can't we turn some of our expertise in
developing methods into methods that students and researchers can use to help them formulate
ideas for research. I've been working on that area pretty intensively for over a decade now -- I
came up with a structured approach that groups can use to map out their ideas on any topic. This
approach, called concept mapping can be used by research teams to help them clarify and map
out the key research issues in an area, to help them operationalize the programs or interventions
or the outcome measures for their study. The concept mapping method isn't the only method
around that might help researchers formulate good research problems and projects. Virtually any
method that's used to help individuals and groups to think more effectively would probably be
useful in research formulation. Some of the methods that might be included in our toolkit for
research formulation might be: brainstorming, brainwriting, nominal group technique, focus
groups, Delphi methods, and facet theory. And then, of course, there are all of the methods for
identifying relevant literature and previous research work. If you know of any techniques or
methods that you think might be useful when formulating the research problem, please feel free
to add a notation -- if there's a relevant Website, please point to it in the notation.

Problem Formulation
"Well begun is half done" --Aristotle, quoting an old proverb

Where do research topics come from?

So how do researchers come up with the idea for a research project? Probably one of the most
common sources of research ideas is the experience of practical problems in the field. Many
researchers are directly engaged in social, health or human service program implementation and
come up with their ideas based on what they see happening around them. Others aren't directly
involved in service contexts, but work with (or survey) people who are in order to learn what
needs to be better understood. Many of the ideas would strike the outsider as silly or worse. For
instance, in health services areas, there is great interest in the problem of back injuries among
nursing staff. It's not necessarily the thing that comes first to mind when we think about the
health care field. But if you reflect on it for a minute longer, it should be obvious that nurses and
nursing staff do an awful lot of lifting in performing their jobs. They lift and push heavy
equipment, and they lift and push oftentimes heavy patients! If 5 or 10 out of every hundred
nursing staff were to strain their backs on average over the period of one year, the costs would be
enormous -- and that's pretty much what's happening. Even minor injuries can result in increased
absenteeism. Major ones can result in lost jobs and expensive medical bills. The nursing industry
figures that this is a problem that costs tens of millions of dollars annually in increased health
care. And, the health care industry has developed a number of approaches, many of them
educational, to try to reduce the scope and cost of the problem. So, even though it might seem
silly at first, many of these practical problems that arise in practice can lead to extensive research
efforts.
Another source for research ideas is the literature in your specific field. Certainly, many
researchers get ideas for research by reading the literature and thinking of ways to extend or
refine previous research. Another type of literature that acts as a source of good research ideas is
the Requests For Proposals (RFPs) that are published by government agencies and some
companies. These RFPs describe some problem that the agency would like researchers to address
-- they are virtually handing the researcher an idea! Typically, the RFP describes the problem
that needs addressing, the contexts in which it operates, the approach they would like you to take
to investigate to address the problem, and the amount they would be willing to pay for such
research. Clearly, there's nothing like potential research funding to get researchers to focus on a
particular research topic.
And let's not forget the fact that many researchers simply think up their research topic on their
own. Of course, no one lives in a vacuum, so we would expect that the ideas you come up with
on your own are influenced by your background, culture, education and experiences.
Is the study feasible?
Very soon after you get an idea for a study reality begins to kick in and you begin to think about
whether the study is feasible at all. There are several major considerations that come into play.
Many of these involve making tradeoffs between rigor and practicality. To do a study well
from a scientific point of view may force you to do things you wouldn't do normally. You may
have to control the implementation of your program more carefully than you otherwise might.
Or, you may have to ask program participants lots of questions that you usually wouldn't if you
weren't doing research. If you had unlimited resources and unbridled control over the
circumstances, you would always be able to do the best quality research. But those ideal
circumstances seldom exist, and researchers are almost always forced to look for the best
tradeoffs they can find in order to get the rigor they desire.
There are several practical considerations that almost always need to be considered when
deciding on the feasibility of a research project. First, you have to think about how long the
research will take to accomplish. Second, you have to question whether there are important
ethical constraints that need consideration. Third, can you achieve the needed cooperation to
take the project to its successful conclusion. And fourth, how significant are the costs of
conducting the research. Failure to consider any of these factors can mean disaster later.
The Literature Review
One of the most important early steps in a research project is the conducting of the literature
review. This is also one of the most humbling experiences you're likely to have. Why? Because
you're likely to find out that just about any worthwhile idea you will have has been thought of
before, at least to some degree. Every time I teach a research methods course, I have at least one
student come to me complaining that they couldn't find anything in the literature that was related
to their topic. And virtually every time they have said that, I was able to show them that was only
true because they only looked for articles that were exactly the same as their research topic. A
literature review is designed to identify related research, to set the current research project within
a conceptual and theoretical context. When looked at that way, there is almost no topic that is so
new or unique that we can't locate relevant and informative related research.
Some tips about conducting the literature review. First, concentrate your efforts on the
scientific literature. Try to determine what the most credible research journals are in your
topical area and start with those. Put the greatest emphasis on research journals that use a blind
review system. In a blind review, authors submit potential articles to a journal editor who solicits
several reviewers who agree to give a critical review of the paper. The paper is sent to these
reviewers with no identification of the author so that there will be no personal bias (either for or
against the author). Based on the reviewers' recommendations, the editor can accept the article,
reject it, or recommend that the author revise and resubmit it. Articles in journals with blind
review processes can be expected to have a fairly high level of credibility. Second, do the
review early in the research process. You are likely to learn a lot in the literature review that will
help you in making the tradeoffs you'll need to face. After all, previous researchers also had to
face tradeoff decisions.
What should you look for in the literature review? First, you might be able to find a study that is
quite similar to the one you are thinking of doing. Since all credible research studies have to
review the literature themselves, you can check their literature review to get a quick-start on your
own. Second, prior research will help assure that you include all of the major relevant constructs
in your study. You may find that other similar studies routinely look at an outcome that you
might not have included. If you did your study without that construct, it would not be judged
credible if it ignored a major construct. Third, the literature review will help you to find and
select appropriate measurement instruments. You will readily see what measurement instruments
researchers use themselves in contexts similar to yours. Finally, the literature review will help
you to anticipate common problems in your research context. You can use the prior experiences
of other to avoid common traps and pitfalls.

Concept Mapping
Social scientists have developed a number of methods and processes that might be useful in
helping you to formulate a research project. I would include among these at least the following --
brainstorming, brainwriting, nominal group techniques, focus groups, affinity mapping, Delphi
techniques, facet theory, and qualitative text analysis. Here, I'll show you a method that I have
developed, called concept mapping, which is especially useful for research problem formulation.
Concept mapping is a general method that can be used to help any individual or group to
describe their ideas about some topic in a pictorial form. There are several different types of
methods that all currently go by names like "concept mapping", "mental mapping" or "concept
webbing." All of them are similar in that they result in a picture of someone's ideas. But the kind
of concept mapping I want to describe here is different in a number of important ways. First, it is
primarily a group process and so it is especially well-suited for situations where teams or groups
of stakeholders have to work together. The other methods work primarily with individuals.
Second, it uses a very structured facilitated approach. There are specific steps that are followed
by a trained facilitator in helping a group to articulate its ideas and understand them more
clearly. Third, the core of concept mapping consists of several state-of-the-art multivariate
statistical methods that analyze the input from all of the individuals and yields an aggregate
group product. And fourth, the method requires the use of specialized computer programs that
can handle the data from this type of process and accomplish the correct analysis and mapping
procedures.
Although concept mapping is a general method, it is particularly useful for helping social
researchers and research teams develop and detail ideas for research. And, it is especially
valuable when researchers want to involve relevant stakeholder groups in the act of creating the
research project. Although concept mapping is used for many purposes -- strategic planning,
product development, market analysis, decision making, measurement development -- we
concentrate here on it's potential for helping researchers formulate their projects.
So what is concept mapping? Essentially, concept mapping is a structured process, focused on
a topic or construct of interest, involving input from one or more participants, that produces
an interpretable pictorial view (concept map) of their ideas and concepts and how these are
interrelated. Concept mapping helps people to think more effectively as a group without losing
their individuality. It helps groups to manage the complexity of their ideas without trivializing
them or losing detail.

A concept mapping process involves six steps that can take place in a single day or can be spread
out over weeks or months depending on the situation. The first step is the Preparation Step.
There are three things done here. The facilitator of the mapping process works with the
initiator(s) (i.e., whoever requests the process initially) to identify who the participants will be. A
mapping process can have hundreds or even thousands of stakeholders participating, although we
usually have a relatively small group of between 10 and 20 stakeholders involved. Second, the
initiator works with the stakeholders to develop the focus for the project. For instance, the group
might decide to focus on defining a program or treatment. Or, they might choose to map all of
the outcomes they might expect to see as a result. Finally, the group decides on an appropriate
schedule for the mapping. In the Generation Step the stakeholders develop a large set of
statements that address the focus. For instance, they might generate statements that describe all
of the specific activities that will constitute a specific social program. Or, they might generate
statements describing specific outcomes that might occur as a result of participating in a
program. A wide variety of methods can be used to accomplish this including traditional
brainstorming, brainwriting, nominal group techniques, focus groups, qualitative text analysis,
and so on. The group can generate up to 200 statements in a concept mapping project. In the
Structuring Step the participants do two things. First, each participant sorts the statements into
piles of similar ones. Most times they do this by sorting a deck of cards that has one statement on
each card. But they can also do this directly on a computer by dragging the statements into piles
that they create. They can have as few or as many piles as they want. Each participant names
each pile with a short descriptive label. Second, each participant rates each of the statements on
some scale. Usually the statements are rated on a 1-to-5 scale for their relative importance, where
a 1 means the statement is relatively unimportant compared to all the rest, a 3 means that it is
moderately important, and a 5 means that it is extremely important. The Representation Step is
where the analysis is done -- this is the process of taking the sort and rating input and
"representing" it in map form. There are two major statistical analyses that are used. The first --
multidimensional scaling -- takes the sort data across all participants and develops the basic map
where each statement is a point on the map and statements that were piled together by more
people are closer to each other on the map. The second analysis -- cluster analysis -- takes the
output of the multidimensional scaling (the point map) and partitions the map into groups of
statements or ideas, into clusters. If the statements describe activities of a program, the clusters
show how these can be grouped into logical groups of activities. If the statements are specific
outcomes, the clusters might be viewed as outcome constructs or concepts. In the fifth step -- the
Interpretation Step -- the facilitator works with the stakeholder group to help them develop
their own labels and interpretations for the various maps. Finally, the Utilization Step involves
using the maps to help address the original focus. On the program side, the maps can be used as a
visual framework for operationalizing the program. on the outcome side, they can be used as the
basis for developing measures and displaying results.
This is only a very basic introduction to concept mapping and its uses. If you want to find out
more about this method, you might look at some of the articles I've written about concept
mapping, including An Introduction to Concept Mapping, Concept Mapping: Soft Science or
Hard Art?,or the article entitled Using Concept Mapping to Develop a Conceptual Framework of
Staff's Views of a Supported Employment Program for Persons with Severe Mental Illness.

Evaluation Research
One specific form of social research -- evaluation research -- is of particular interest here. The
Introduction to Evaluation Research presents an overview of what evaluation is and how it
differs from social research generally. We also introduce several evaluation models to give you
some perspective on the evaluation endeavor. Evaluation should not be considered in a vacuum.
Here, we consider evaluation as embedded within a larger Planning-Evaluation Cycle.
Evaluation can be a threatening activity. Many groups and organizations struggle with how to
build a good evaluation capability into their everyday activities and procedures. This is
essentially an organizational culture issue. Here we consider some of the issues a group or
organization needs to address in order to develop an evaluation culture that works in their
context.
Introduction to Evaluation
Evaluation is a methodological area that is closely related to, but distinguishable from more
traditional social research. Evaluation utilizes many of the same methodologies used in
traditional social research, but because evaluation takes place within a political and
organizational context, it requires group skills, management ability, political dexterity, sensitivity
to multiple stakeholders and other skills that social research in general does not rely on as much.
Here we introduce the idea of evaluation and some of the major terms and issues in the field.
Definitions of Evaluation
Probably the most frequently given definition is:
Evaluation is the systematic assessment of the worth or merit of some object
This definition is hardly perfect. There are many types of evaluations that do not necessarily
result in an assessment of worth or merit -- descriptive studies, implementation analyses, and
formative evaluations, to name a few. Better perhaps is a definition that emphasizes the
information-processing and feedback functions of evaluation. For instance, one might say:
Evaluation is the systematic acquisition and assessment of information to provide useful
feedback about some object
Both definitions agree that evaluation is a systematic endeavor and both use the deliberately
ambiguous term 'object' which could refer to a program, policy, technology, person, need,
activity, and so on. The latter definition emphasizes acquiring and assessing information rather
than assessing worth or merit because all evaluation work involves collecting and sifting through
data, making judgements about the validity of the information and of inferences we derive from
it, whether or not an assessment of worth or merit results.
The Goals of Evaluation
The generic goal of most evaluations is to provide "useful feedback" to a variety of audiences
including sponsors, donors, client-groups, administrators, staff, and other relevant constituencies.
Most often, feedback is perceived as "useful" if it aids in decision-making. But the relationship
between an evaluation and its impact is not a simple one -- studies that seem critical sometimes
fail to influence short-term decisions, and studies that initially seem to have no influence can
have a delayed impact when more congenial conditions arise. Despite this, there is broad
consensus that the major goal of evaluation should be to influence decision-making or policy
formulation through the provision of empirically-driven feedback.
Evaluation Strategies
'Evaluation strategies' means broad, overarching perspectives on evaluation. They encompass the
most general groups or "camps" of evaluators; although, at its best, evaluation work borrows
eclectically from the perspectives of all these camps. Four major groups of evaluation strategies
are discussed here.
Scientific-experimental models are probably the most historically dominant evaluation
strategies. Taking their values and methods from the sciences -- especially the social sciences --
they prioritize on the desirability of impartiality, accuracy, objectivity and the validity of the
information generated. Included under scientific-experimental models would be: the tradition of
experimental and quasi-experimental designs; objectives-based research that comes from
education; econometrically-oriented perspectives including cost-effectiveness and cost-benefit
analysis; and the recent articulation of theory-driven evaluation.
The second class of strategies are management-oriented systems models. Two of the most
common of these are PERT, the Program Evaluation and Review Technique, and CPM, the
Critical Path Method. Both have been widely used in business and government in this country. It
would also be legitimate to include the Logical Framework or "Logframe" model developed at
U.S. Agency for International Development and general systems theory and operations research
approaches in this category. Two management-oriented systems models were originated by
evaluators: the UTOS model where U stands for Units, T for Treatments, O for Observing
Observations and S for Settings; and the CIPP model where the C stands for Context, the I for
Input, the first P for Process and the second P for Product. These management-oriented systems
models emphasize comprehensiveness in evaluation, placing evaluation within a larger
framework of organizational activities.
The third class of strategies are the qualitative/anthropological models. They emphasize the
importance of observation, the need to retain the phenomenological quality of the evaluation
context, and the value of subjective human interpretation in the evaluation process. Included in
this category are the approaches known in evaluation as naturalistic or 'Fourth Generation'
evaluation; the various qualitative schools; critical theory and art criticism approaches; and, the
'grounded theory' approach of Glaser and Strauss among others.
Finally, a fourth class of strategies is termed participant-oriented models. As the term suggests,
they emphasize the central importance of the evaluation participants, especially clients and users
of the program or technology. Client-centered and stakeholder approaches are examples of
participant-oriented models, as are consumer-oriented evaluation systems.
With all of these strategies to choose from, how to decide? Debates that rage within the
evaluation profession -- and they do rage -- are generally battles between these different
strategists, with each claiming the superiority of their position. In reality, most good evaluators
are familiar with all four categories and borrow from each as the need arises. There is no inherent
incompatibility between these broad strategies -- each of them brings something valuable to the
evaluation table. In fact, in recent years attention has increasingly turned to how one might
integrate results from evaluations that use different strategies, carried out from different
perspectives, and using different methods. Clearly, there are no simple answers here. The
problems are complex and the methodologies needed will and should be varied.
Types of Evaluation
There are many different types of evaluations depending on the object being evaluated and the
purpose of the evaluation. Perhaps the most important basic distinction in evaluation types is that
between formative and summative evaluation. Formative evaluations strengthen or improve the
object being evaluated -- they help form it by examining the delivery of the program or
technology, the quality of its implementation, and the assessment of the organizational context,
personnel, procedures, inputs, and so on. Summative evaluations, in contrast, examine the effects
or outcomes of some object -- they summarize it by describing what happens subsequent to
delivery of the program or technology; assessing whether the object can be said to have caused
the outcome; determining the overall impact of the causal factor beyond only the immediate
target outcomes; and, estimating the relative costs associated with the object.
Formative evaluation includes several evaluation types:
 • needs assessment determines who needs the program, how great the need
is, and what might work to meet the need
• evaluability assessment determines whether an evaluation is feasible and
how stakeholders can help shape its usefulness
• structured conceptualization helps stakeholders define the program or
technology, the target population, and the possible outcomes
• implementation evaluation monitors the fidelity of the program or
technology delivery
• process evaluation investigates the process of delivering the program or
technology, including alternative delivery procedures
Summative evaluation can also be subdivided:
• outcome evaluations investigate whether the program or technology
caused demonstrable effects on specifically defined target outcomes
• impact evaluation is broader and assesses the overall or net effects --
intended or unintended -- of the program or technology as a whole
• cost-effectiveness and cost-benefit analysis address questions of
efficiency by standardizing outcomes in terms of their dollar costs and values
• secondary analysis reexamines existing data to address new questions or
use methods not previously employed
• meta-analysis integrates the outcome estimates from multiple studies to
arrive at an overall or summary judgement on an evaluation question

Evaluation Questions and Methods

Evaluators ask many different kinds of questions and use a variety of methods to address them.
These are considered within the framework of formative and summative evaluation as presented
above.
In formative research the major questions and methodologies are:
What is the definition and scope of the problem or issue, or what's the question?
Formulating and conceptualizing methods might be used including brainstorming, focus groups,
nominal group techniques, Delphi methods, brainwriting, stakeholder analysis, synectics, lateral
thinking, input-output analysis, and concept mapping.
Where is the problem and how big or serious is it?
The most common method used here is "needs assessment" which can include: analysis of
existing data sources, and the use of sample surveys, interviews of constituent populations,
qualitative research, expert testimony, and focus groups.
How should the program or technology be delivered to address the problem?
Some of the methods already listed apply here, as do detailing methodologies like simulation
techniques, or multivariate methods like multiattribute utility theory or exploratory causal
modeling; decision-making methods; and project planning and implementation methods like
flow charting, PERT/CPM, and project scheduling.
How well is the program or technology delivered?
Qualitative and quantitative monitoring techniques, the use of management information systems,
and implementation assessment would be appropriate methodologies here.
The questions and methods addressed under summative evaluation
include:
What type of evaluation is feasible?
Evaluability assessment can be used here, as well as standard approaches for selecting an
appropriate evaluation design.
What was the effectiveness of the program or technology?
One would choose from observational and correlational methods for demonstrating whether
desired effects occurred, and quasi-experimental and experimental designs for determining
whether observed effects can reasonably be attributed to the intervention and not to other
sources.
What is the net impact of the program?
Econometric methods for assessing cost effectiveness and cost/benefits would apply here, along
with qualitative methods that enable us to summarize the full range of intended and unintended
impacts.
Clearly, this introduction is not meant to be exhaustive. Each of these methods, and the many not
mentioned, are supported by an extensive methodological research literature. This is a
formidable set of tools. But the need to improve, update and adapt these methods to changing
circumstances means that methodological research and development needs to have a major place
in evaluation work.

The Planning-Evaluation Cycle

Often, evaluation is
construed as part of
a larger managerial
or administrative
process.
Sometimes this is
referred to as the
planning-
evaluation cycle.
The distinctions
between planning
and evaluation are
not always clear;
this cycle is
described in many
different ways with
various phases
claimed by both
planners and
evaluators.
Usually, the first
stage of such a cycle -- the planning phase -- is designed to elaborate a set of potential actions,
programs, or technologies, and select the best for implementation. Depending on the
organization and the problem being addressed, a planning process could involve any or all of
these stages: the formulation of the problem, issue, or concern; the broad conceptualization of
the major alternatives that might be considered; the detailing of these alternatives and their
potential implications; the evaluation of the alternatives and the selection of the best one; and the
implementation of the selected alternative. Although these stages are traditionally considered
planning, there is a lot of evaluation work involved. Evaluators are trained in needs assessment,
they use methodologies -- like the concept mapping one presented later -- that help in
conceptualization and detailing, and they have the skills to help assess alternatives and make a
choice of the best one.
The evaluation phase also involves a sequence of stages that typically includes: the formulation
of the major objectives, goals, and hypotheses of the program or technology; the
conceptualization and operationalization of the major components of the evaluation -- the
program, participants, setting, and measures; the design of the evaluation, detailing how these
components will be coordinated; the analysis of the information, both qualitative and
quantitative; and the utilization of the evaluation results.

An Evaluation Culture
I took the idea of an evaluation culture from a wonderful paper written by Donald Campbell in
1969 entitled 'Methods for an Experimenting Society.' Following in the footsteps of that paper,
this one is considerably more naive and utopian. And, I have changed the name of this idealized
society to reflect terminology that is perhaps more amenable to the climate of the 1990s. For the
term experimenting, I have substituted the softer and broader term evaluating. And for the term
society, I have substituted the more internationally-flavored term culture. With these shifts in
emphasis duly noted, I want you to know that I see the evaluation culture as one that a member
of the experimenting society would feel comfortable visiting, and perhaps even thinking of
taking as a permanent residence.
What would an evaluation culture look like? What should its values be? You should know at the
outset that I fully hope that some version of this fantasy will become an integral part of twenty-
first century thought. There is no particular order of importance to the way these ideas are
presented -- I'll leave that ordering to subsequent efforts.
First, our evaluation culture will embrace an action-oriented perspective that actively seeks
solutions to problems, trying out tentative ones, weighing the results and consequences of
actions, all within an endless cycle of supposition-action-evidence-revision that characterizes
good science and good management. In this activist evaluation culture, we will encourage
innovative approaches at all levels. But well-intentioned activism by itself is not enough, and
may at times be risky, dangerous, and lead to detrimental consequences. In an evaluation culture,
we won't act for action's sake -- we'll always attempt to assess the effects of our actions.
This evaluation culture will be an accessible, teaching-oriented one that emphasizes the unity of
formal evaluation and everyday thought. Most of our evaluations will be simple, informal,
efficient, practical, low-cost and easily carried out and understood by nontechnicians.
Evaluations won't just be delegated to one person or department -- we will encourage everyone
in our organizations to become involved in evaluating what they and their organizations do.
Where technical expertise is needed we will encourage the experts to also educate us about the
technical side of what they do, demanding that they try to find ways to explain their techniques
and methods adequately for nontechnicians. We will devote considerable resources to teaching
others about evaluation principles.
Our evaluation culture will be diverse, inclusive, participatory, responsive and fundamentally
non-hierarchical. World problems cannot be solved by simple "silver bullet" solutions. There is
growing recognition in many arenas that our most fundamental problems are systemic,
interconnected, and inextricably linked to social and economic issues and factors. Solutions will
involve husbanding the resources, talents and insights of a wide range of people. The formulation
of problems and potential solutions needs to involve a broad range of constituencies. More than
just "research" skills will be needed. Especially important will be skills in negotiation and
consensus-building processes. Evaluators are familiar with arguments for greater diversity and
inclusiveness -- we've been talking about stakeholder, participative, multiple-constituency
research for nearly two decades. No one that I know is seriously debating anymore whether we
should move to more inclusive participatory approaches. The real question seems to be how such
work might best be accomplished, and despite all the rhetoric about the importance of
participatory methods, we have a long way to go in learning how to do them effectively.
Our evaluation culture will be a humble, self-critical one. We will openly acknowledge our
limitations and recognize that what we learn from a single evaluation study, however well
designed, will almost always be equivocal and tentative. In this regard, I believe we too often
undervalue cowardice in research. I find it wholly appropriate that evaluators resist being drawn
into making decisions for others, although certainly the results of our work should help inform
the decision makers. A cowardly approach saves the evaluator from being drawn into the
political context, helping assure the impartiality needed for objective assessment, and it protects
the evaluator from taking responsibility for making decisions that should be left to those who
have been duly-authorized -- and who have to live with the consequences. Most program
decisions, especially decisions about whether to continue a program or close it down, must
include more input than an evaluation alone can ever provide. While evaluators can help to
elucidate what has happened in the past or might happen under certain circumstances, it is the
responsibility of the organization and society as a whole to determine what ought to happen. The
debate about the appropriate role of an evaluator in the decision-making process is an extremely
intense one right now in evaluation circles, and my position advocating a cowardly reluctance of
the evaluator to undertake a decision-making role may very well be in the minority. We will
need to debate this issue vigorously, especially for politically-complex, international-evaluation
contexts.
Our evaluation culture will need to be an interdisciplinary one, doing more than just grafting one
discipline onto another through constructing multi-discipline research teams. We'll need such
teams, of course, but I mean to imply something deeper, more personally internalized -- we need
to move toward being nondisciplinary, consciously putting aside the blinders of our respective
specialties in an attempt to foster a more whole view of the phenomena we study. As we consider
the programs we are evaluating, we each should be able to speculate about a broad range of
implementation factors or potential consequences. We should be able to anticipate some of the
organizational and systems-related features of these programs, the economic factors that might
enhance or reduce implementation, their social and psychological dimensions, and especially
whether the ultimate utilizers can understand or know how to utilize and be willing to utilize the
the results of our evaluation work. We should also be able to anticipate a broad spectrum of
potential consequences -- system-related, production-related, economic, nutritional, social,
environmental.
This evaluation culture will also be an honest, truth-seeking one that stresses accountability and
scientific credibility. In many quarters in contemporary society, it appears that many people have
given up on the ideas of truth and validity. Our evaluation culture needs to hold to the goal of
getting at the truth while at the same time honestly acknowledging the revisability of all
scientific knowledge. We need to be critical of those who have given up on the goal of "getting it
right" about reality, especially those among the humanities and social sciences who argue that
truth is entirely relative to the knower, objectivity an impossibility, and reality nothing more than
a construction or illusion that cannot be examined publicly. For them, the goal of seeking the
truth is inappropriate and unacceptable, and science a tool of oppression rather than a road to
greater enlightenment. Philosophers have, of course, debated such issues for thousands of years
and will undoubtedly do so for thousands more. We in the evaluation culture need to check in on
their thinking from time to time, but until they settle these debates, we need to hold steadfastly to
the goal of getting at the truth -- the goal of getting it right about reality.
Our evaluation culture will be prospective and forward-looking, anticipating where evaluation
feedback will be needed rather than just reacting to situations as they arise. We will construct
simple, low-cost evaluation and monitoring information systems when we first initiate a new
program or technology -- we cannot wait until a program is complete or a technology is in the
field before we turn our attention to its evaluation.
Finally, the evaluation culture I envision is one that will emphasize fair, open, ethical and
democratic processes. We should move away from private ownership of and exclusive access to
data. The data from all of our evaluations needs to be accessible to all interested groups allowing
more extensive independent secondary analyses and opportunities for replication or refutation of
original results. We should encourage open commentary and debate regarding the results of
specific evaluations. Especially when there are multiple parties who have a stake in such results,
it is important for our reporting procedures to include formal opportunities for competitive
review and response. Our evaluation culture must continually strive for greater understanding of
the ethical dilemmas posed by our research. Our desire for valid, scientific inference will at times
put us in conflict with ethical principles. The situation is likely to be especially complex in
international-evaluation contexts where we will often be dealing with multiple cultures and
countries that are at different stages of economic development and have different value systems
and morals. We need to be ready to deal with potential ethical and political issues posed by our
methodologies in an open, direct, and democratic manner.
Do you agree with the values I'm describing here? What other characteristics might this
evaluation culture have? You tell me. There are many more values and characteristics that ought
to be considered. For now, the ones mentioned above, and others in the literature, provide us
with a starting point at which we can all join the discussion. I hope you will add to the list, and I
encourage each of you to criticize these tentative statements I've offered about the extraordinary
potential of the evaluation culture that we are all in the process of creating today.

Sampling
Sampling is the process of selecting units (e.g., people, organizations) from a population of
interest so that by studying the sample we may fairly generalize our results back to the
population from which they were chosen. Let's begin by covering some of the key terms in
sampling like "population" and "sampling frame." Then, because some types of sampling rely
upon quantitative models, we'll talk about some of the statistical terms used in sampling. Finally,
we'll discuss the major distinction between probability and Nonprobability sampling methods
and work through the major types in each.

External Validity
External validity is related to generalizing. That's the major thing you need to keep in mind.
Recall that validity refers to the approximate truth of propositions, inferences, or conclusions.
So, external validity refers to the approximate truth of conclusions the involve generalizations.
Put in more pedestrian terms, external validity is the degree to which the conclusions in your
study would hold for other persons in other places and at other times.
In science there are
two major
approaches to how
we provide
evidence for a
generalization. I'll
call the first
approach the
Sampling Model.
In the sampling
model, you start by
identifying the
population you
would like to
generalize to.
Then, you draw a
fair sample from
that population and
conduct your research with the sample. Finally, because the sample is representative of the
population, you can automatically generalize your results back to the population. There are
several problems with this approach. First, perhaps you don't know at the time of your study who
you might ultimately like to generalize to. Second, you may not be easily able to draw a fair or
representative sample. Third, it's impossible to sample across all times that you might like to
generalize to (like next year).
I'll call the second
approach to
generalizing the
Proximal
Similarity Model.
'Proximal' means
'nearby' and
'similarity' means...
well, it means
'similarity'. The
term proximal
similarity was
suggested by
Donald T.
Campbell as an
appropriate
relabeling of the
term external validity (although he was the first to admit that it probably wouldn't catch on!).
Under this model, we begin by thinking about different generalizability contexts and developing
a theory about which contexts are more like our study and which are less so. For instance, we
might imagine several settings that have people who are more similar to the people in our study
or people who are less similar. This also holds for times and places. When we place different
contexts in terms of their relative similarities, we can call this implicit theoretical a gradient of
similarity. Once we have developed this proximal similarity framework, we are able to
generalize. How? We conclude that we can generalize the results of our study to other persons,
places or times that are more like (that is, more proximally similar) to our study. Notice that here,
we can never generalize with certainty -- it is always a question of more or less similar.
Threats to External Validity
A threat to external validity is an explanation of how you might be wrong in making a
generalization. For instance, you conclude that the results of your study (which was done in a
specific place, with certain types of people, and at a specific time) can be generalized to another
context (for instance, another place, with slightly different people, at a slightly later time). There
are three major threats to external validity because there are three ways you could be wrong --
people, places or times. Your critics could come along, for example, and argue that the results of
your study are due to the unusual type of people who were in the study. Or, they could argue that
it might only work because of the unusual place you did the study in (perhaps you did your
educational study in a college town with lots of high-achieving educationally-oriented kids). Or,
they might suggest that you did your study in a peculiar time. For instance, if you did your
smoking cessation study the week after the Surgeon General issues the well-publicized results of
the latest smoking and cancer studies, you might get different results than if you had done it the
week before.
Improving External Validity
How can we improve external validity? One way, based on the sampling model, suggests that
you do a good job of drawing a sample from a population. For instance, you should use random
selection, if possible, rather than a nonrandom procedure. And, once selected, you should try to
assure that the respondents participate in your study and that you keep your dropout rates low. A
second approach would be to use the theory of proximal similarity more effectively. How?
Perhaps you could do a better job of describing the ways your contexts and others differ,
providing lots of data about the degree of similarity between various groups of people, places,
and even times. You might even be able to map out the degree of proximal similarity among
various contexts with a methodology like concept mapping. Perhaps the best approach to
criticisms of generalizations is simply to show them that they're wrong -- do your study in a
variety of places, with different people and at different times. That is, your external validity
(ability to generalize) will be stronger the more you replicate your study.

Sampling Terminology
As with anything else in life you have to learn the language of an area if you're going to ever
hope to use it. Here, I want to introduce several different terms for the major groups that are
involved in a sampling process and the role that each group plays in the logic of sampling.
The major question that motivates sampling in the first place is: "Who do you want to generalize
to?" Or should it be: "To whom do you want to generalize?" In most social research we are
interested in more than just the people who directly participate in our study. We would like to be
able to talk in general terms and not be confined only to the people who are in our study. Now,
there are times when we aren't very concerned about generalizing. Maybe we're just evaluating a
program in a local agency and we don't care whether the program would work with other people
in other places and at other times. In that case, sampling and generalizing might not be of
interest. In other cases, we would really like to be able to generalize almost universally. When
psychologists do research, they are often interested in developing theories that would hold for all
humans. But in most applied social research, we are interested in generalizing to specific groups.
The group you wish to generalize to is often called the population in your study. This is the
group you would like to sample from because this is the group you are interested in generalizing
to. Let's imagine that you wish to generalize to urban homeless males between the ages of 30 and
50 in the United States. If that is the population of interest, you are likely to have a very hard
time developing a reasonable sampling plan. You are probably not going to find an accurate
listing of this population, and even if you did, you would almost certainly not be able to mount a
national sample across hundreds of urban areas. So we probably should make a distinction
between the population you would like to generalize to, and the population that will be accessible
to you. We'll call the former the theoretical population and the latter the accessible population.
In this example, the accessible population might be homeless males between the ages of 30 and
50 in six selected urban areas across the U.S.
Once you've identified the theoretical and accessible populations, you have to do one more thing
before you can actually draw a sample -- you have to get a list of the members of the accessible
population. (Or, you have to spell out in detail how you will contact them to assure
representativeness). The listing of the accessible population from which you'll draw your sample
is called the sampling frame. If you were doing a phone survey and selecting names from the
telephone book, the book would be your sampling frame. That wouldn't be a great way to sample
because significant subportions of the population either don't have a phone or have moved in or
out of the area since the last book was printed. Notice that in this case, you might identify the
area code and all three-digit prefixes within that area code and draw a sample simply by
randomly dialing numbers (cleverly known as random-digit-dialing). In this case, the sampling
frame is not a list per se, but is rather a procedure that you follow as the actual basis for
sampling. Finally, you actually draw your sample (using one of the many sampling procedures).
The sample is the group of people who you select to be in your study. Notice that I didn't say
that the sample was the group of people who are actually in your study. You may not be able to
contact or recruit all of the people you actually sample, or some could drop out over the course
of the study. The group that actually completes your study is a subsample of the sample -- it
doesn't include nonrespondents or dropouts. The problem of nonresponse and its effects on a
study will be addressed when discussing "mortality" threats to internal validity.
People often confuse what is meant by random selection with the idea of random assignment.
You should make sure that you understand the distinction between random selection and random
assignment.
At this point, you should appreciate that sampling is a difficult multi-step process and that there
are lots of places you can go wrong. In fact, as we move from each step to the next in identifying
a sample, there is the possibility of introducing systematic error or bias. For instance, even if you
are able to identify perfectly the population of interest, you may not have access to all of them.
And even if you do, you may not have a complete and accurate enumeration or sampling frame
from which to select. And, even if you do, you may not draw the sample correctly or accurately.
And, even if you do, they may not all come and they may not all stay. Depressed yet? This is a
very difficult business indeed. At times like this I'm reminded of what Donald Campbell used to
say (I'll paraphrase here): "Cousins to the amoeba, it's amazing that we know anything at all!"

Statistical Terms in Sampling

Let's begin by
defining some very
simple terms that
are relevant here.
First, let's look at
the results of our
sampling efforts.
When we sample,
the units that we
sample -- usually
people -- supply us
with one or more
responses. In this
sense, a response
is a specific
measurement value
that a sampling
unit supplies. In
the figure, the
person is
responding to a
survey instrument
and gives a
response of '4'.
When we look
across the
responses that we get for our entire sample, we use a statistic. There are a wide variety of
statistics we can use -- mean, median, mode, and so on. In this example, we see that the mean or
average for the sample is 3.75. But the reason we sample is so that we might get an estimate for
the population we sampled from. If we could, we would much prefer to measure the entire
population. If you measure the entire population and calculate a value like a mean or average, we
don't refer to this as a statistic, we call it a parameter of the population.
The Sampling Distribution
So how do we get from our sample statistic to an estimate of the population parameter? A crucial
midway concept you need to understand is the sampling distribution. In order to understand it,
you have to be able and willing to do a thought experiment. Imagine that instead of just taking a
single sample like we do in a typical study, you took three independent samples of the same
population. And furthermore, imagine that for each of your three samples, you collected a single
response and computed a single statistic, say, the mean of the response. Even though all three
samples came from the same population, you wouldn't expect to get the exact same statistic from
each. They would differ slightly just due to the random "luck of the draw" or to the natural
fluctuations or vagaries of drawing a sample. But you would expect that all three samples would
 yield a similar
statistical estimate
because they were
drawn from the
same population.
Now, for the leap
of imagination!
Imagine that you
did an infinite
number of samples
from the same
population and
computed the
average for each
one. If you plotted
them on a
histogram or bar
graph you should
find that most of
them converge on
the same central value and that you get fewer and fewer samples that have averages farther away
up or down from that central value. In other words, the bar graph would be well described by the
bell curve shape that is an indication of a "normal" distribution in statistics. The distribution of
an infinite number of samples of the same size as the sample in your study is known as the
sampling distribution. We don't ever actually construct a sampling distribution. Why not?
You're not paying attention! Because to construct it we would have to take an infinite number of
samples and at least the last time I checked, on this planet infinite is not a number we know how
to reach. So why do we even talk about a sampling distribution? Now that's a good question!
Because we need to realize that our sample is just one of a potentially infinite number of samples
that we could have taken. When we keep the sampling distribution in mind, we realize that while
the statistic we got from our sample is probably near the center of the sampling distribution
(because most of the samples would be there) we could have gotten one of the extreme samples
just by the luck of the draw. If we take the average of the sampling distribution -- the average of
the averages of an infinite number of samples -- we would be much closer to the true population
average -- the parameter of interest. So the average of the sampling distribution is essentially
equivalent to the parameter. But what is the standard deviation of the sampling distribution (OK,
never had statistics? There are any number of places on the web where you can learn about them
or even just brush up if you've gotten rusty. This isn't one of them. I'm going to assume that you
at least know what a standard deviation is, or that you're capable of finding out relatively
quickly). The standard deviation of the sampling distribution tells us something about how
different samples would be distributed. In statistics it is referred to as the standard error (so we
can keep it separate in our minds from standard deviations. Getting confused? Go get a cup of
coffee and come back in ten minutes...OK, let's try once more... A standard deviation is the
spread of the scores around the average in a single sample. The standard error is the spread of the
averages around the average of averages in a sampling distribution. Got it?)
Sampling Error
In sampling contexts, the standard error is called sampling error. Sampling error gives us some
idea of the precision of our statistical estimate. A low sampling error means that we had
relatively less variability or range in the sampling distribution. But here we go again -- we never
actually see the sampling distribution! So how do we calculate sampling error? We base our
calculation on the standard deviation of our sample. The greater the sample standard deviation,
the greater the standard error (and the sampling error). The standard error is also related to the
sample size. The greater your sample size, the smaller the standard error. Why? Because the
greater the sample size, the closer your sample is to the actual population itself. If you take a
sample that consists of the entire population you actually have no sampling error because you
don't have a sample, you have the entire population. In that case, the mean you estimate is the
parameter.
The 68, 95,
99 Percent
Rule
You've probably
heard this one
before, but it's so
important that it's
always worth
repeating... There is
a general rule that
applies whenever we
have a normal or
bell-shaped
distribution. Start
with the average --
the center of the
distribution. If you
go up and down (i.e.,
left and right) one
standard unit, you
will include
approximately 68%
of the cases in the
distribution (i.e.,
68% of the area
under the curve). If
you go up and down two standard units, you will include approximately 95% of the cases. And if
you go plus-and-minus three standard units, you will include about 99% of the cases. Notice that
I didn't specify in the previous few sentences whether I was talking about standard deviation
units or standard error units. That's because the same rule holds for both types of distributions
(i.e., the raw data and sampling distributions). For instance, in the figure, the mean of the
distribution is 3.75 and the standard unit is .25 (If this was a distribution of raw data, we would
be talking in standard deviation units. If it's a sampling distribution, we'd be talking in standard
error units). If we go up and down one standard unit from the mean, we would be going up and
down .25 from the mean of 3.75. Within this range -- 3.5 to 4.0 -- we would expect to see
approximately 68% of the cases. This section is marked in red on the figure. I leave to you to
figure out the other ranges. But what does this all mean you ask? If we are dealing with raw data
and we know the mean and standard deviation of a sample, we can predict the intervals within
which 68, 95 and 99% of our cases would be expected to fall. We call these intervals the -- guess
what -- 68, 95 and 99% confidence intervals.
Now, here's where
everything should
come together in
one great aha!
experience if
you've been
following along. If
we had a sampling
distribution, we
would be able to
predict the 68, 95
and 99%
confidence
intervals for where
the population
parameter should
be! And isn't that
why we sampled in
the first place? So
that we could
predict where the
population is on
that variable?
There's only one
hitch. We don't
actually have the
sampling
distribution (now this is the third time I've said this in this essay)! But we do have the
distribution for the sample itself. And we can from that distribution estimate the standard error
(the sampling error) because it is based on the standard deviation and we have that. And, of
course, we don't actually know the population parameter value -- we're trying to find that out --
but we can use our best estimate for that -- the sample statistic. Now, if we have the mean of the
sampling distribution (or set it to the mean from our sample) and we have an estimate of the
standard error (we calculate that from our sample) then we have the two key ingredients that we
need for our sampling distribution in order to estimate confidence intervals for the population
parameter.
Perhaps an example will help. Let's assume we did a study and drew a single sample from the
population. Furthermore, let's assume that the average for the sample was 3.75 and the standard
deviation was .25. This is the raw data distribution depicted above. now, what would the
sampling distribution be in this case? Well, we don't actually construct it (because we would
need to take an infinite number of samples) but we can estimate it. For starters, we assume that
the mean of the sampling distribution is the mean of the sample, which is 3.75. Then, we
calculate the standard error. To do this, we use the standard deviation for our sample and the
sample size (in this case N=100) and we come up with a standard error of .025 (just trust me on
this). Now we have everything we need to estimate a confidence interval for the population
parameter. We would estimate that the probability is 68% that the true parameter value falls
between 3.725 and 3.775 (i.e., 3.75 plus and minus .025); that the 95% confidence interval is
3.700 to 3.800; and that we can say with 99% confidence that the population value is between
3.675 and 3.825. The real value (in this fictitious example) was 3.72 and so we have correctly
estimated that value with our sample.

Probability Sampling
A probability sampling method is any method of sampling that utilizes some form of random
selection. In order to have a random selection method, you must set up some process or
procedure that assures that the different units in your population have equal probabilities of being
chosen. Humans have long practiced various forms of random selection, such as picking a name
out of a hat, or choosing the short straw. These days, we tend to use computers as the mechanism
for generating random numbers as the basis for random selection.
Some Definitions
Before I can explain the various probability methods we have to define some basic terms. These
are:
• N = the number of cases in the sampling frame
• n = the number of cases in the sample
• NCn = the number of combinations (subsets) of n from N
• f = n/N = the sampling fraction
That's it. With those terms defined we can begin to define the different probability sampling
methods.
Simple Random Sampling
The simplest form of random sampling is called simple random sampling. Pretty tricky, huh?
Here's the quick description of simple random sampling:
• Objective: To select n units out of N such that each NCn has an equal chance of being
selected.
• Procedure: Use a table of random numbers, a computer random number generator, or a
mechanical device to select the sample.
A somewhat stilted, if accurate,
definition. Let's see if we can
make it a little more real. How do
we select a simple random
sample? Let's assume that we are
doing some research with a small
service agency that wishes to
assess client's views of quality of
service over the past year. First,
we have to get the sampling frame
organized. To accomplish this, we'll go through agency records to identify every client over the
past 12 months. If we're lucky, the agency has good accurate computerized records and can
quickly produce such a list. Then, we have to actually draw the sample. Decide on the number of
clients you would like to have in the final sample. For the sake of the example, let's say you want
to select 100 clients to survey and that there were 1000 clients over the past 12 months. Then, the
sampling fraction is f = n/N = 100/1000 = .10 or 10%. Now, to actually draw the sample, you
have several options. You could print off the list of 1000 clients, tear then into separate strips,
put the strips in a hat, mix them up real good, close your eyes and pull out the first 100. But this
mechanical procedure would be tedious and the quality of the sample would depend on how
thoroughly you mixed them up and how randomly you reached in. Perhaps a better procedure
would be to use the kind of ball machine that is popular with many of the state lotteries. You
would need three sets of balls numbered 0 to 9, one set for each of the digits from 000 to 999 (if
we select 000 we'll call that 1000). Number the list of names from 1 to 1000 and then use the ball
machine to select the three digits that selects each person. The obvious disadvantage here is that
you need to get the ball machines. (Where do they make those things, anyway? Is there a ball
machine industry?).
Neither of these mechanical procedures is very feasible and, with the development of
inexpensive computers there is a much easier way. Here's a simple procedure that's especially
useful if you have the names of the clients already on the computer. Many computer programs
can generate a series of random numbers. Let's assume you can copy and paste the list of client
names into a column in an EXCEL spreadsheet. Then, in the column right next to it paste the
function =RAND() which is EXCEL's way of putting a random number between 0 and 1 in the
cells. Then, sort both columns -- the list of names and the random number -- by the random
numbers. This rearranges the list in random order from the lowest to the highest random number.
Then, all you have to do is take the first hundred names in this sorted list. pretty simple. You
could probably accomplish the whole thing in under a minute.
Simple random sampling is simple to accomplish and is easy to explain to others. Because
simple random sampling is a fair way to select a sample, it is reasonable to generalize the results
from the sample back to the population. Simple random sampling is not the most statistically
efficient method of sampling and you may, just because of the luck of the draw, not get good
representation of subgroups in a population. To deal with these issues, we have to turn to other
sampling methods.
Stratified Random Sampling
Stratified Random Sampling, also sometimes called proportional or quota random sampling,
involves dividing your population into homogeneous subgroups and then taking a simple random
sample in each subgroup. In more formal terms:
Objective: Divide the population into non-overlapping groups (i.e., strata) N1, N2, N3, ... Ni, such
that N1 + N2 + N3 + ... + Ni = N. Then do a simple random sample of f = n/N in each strata.
There are several major reasons why you might prefer stratified sampling over simple random
sampling. First, it assures that you will be able to represent not only the overall population, but
also key subgroups of the population, especially small minority groups. If you want to be able to
talk about subgroups, this may be the only way to effectively assure you'll be able to. If the
subgroup is extremely small, you can use different sampling fractions (f) within the different
strata to randomly over-sample the small group (although you'll then have to weight the within-
group estimates using the sampling fraction whenever you want overall population estimates).
When we use the same sampling fraction within strata we are conducting proportionate stratified
random sampling. When we use different sampling fractions in the strata, we call this
disproportionate stratified random sampling. Second, stratified random sampling will generally
have more statistical precision than simple random sampling. This will only be true if the strata
or groups are homogeneous. If they are, we expect that the variability within-groups is lower
than the variability for the population as a whole. Stratified sampling capitalizes on that fact.
For example, let's
say that the
population of
clients for our
agency can be
divided into three
groups: Caucasian,
African-American
and Hispanic-
American.
Furthermore, let's
assume that both
the African-
Americans and
Hispanic-
Americans are
relatively small
minorities of the
clientele (10% and 5% respectively). If we just did a simple random sample of n=100 with a
sampling fraction of 10%, we would expect by chance alone that we would only get 10 and 5
persons from each of our two smaller groups. And, by chance, we could get fewer than that! If
we stratify, we can do better. First, let's determine how many people we want to have in each
group. Let's say we still want to take a sample of 100 from the population of 1000 clients over
the past year. But we think that in order to say anything about subgroups we will need at least 25
cases in each group. So, let's sample 50 Caucasians, 25 African-Americans, and 25 Hispanic-
Americans. We know that 10% of the population, or 100 clients, are African-American. If we
randomly sample 25 of these, we have a within-stratum sampling fraction of 25/100 = 25%.
Similarly, we know that 5% or 50 clients are Hispanic-American. So our within-stratum
sampling fraction will be 25/50 = 50%. Finally, by subtraction we know that there are 850
Caucasian clients. Our within-stratum sampling fraction for them is 50/850 = about 5.88%.
Because the groups are more homogeneous within-group than across the population as a whole,
we can expect greater statistical precision (less variance). And, because we stratified, we know
we will have enough cases from each group to make meaningful subgroup inferences.
Systematic Random Sampling
Here are the steps you need to follow in order to achieve a systematic random sample:
• number the units in the population from 1 to N
• decide on the n (sample size) that you want or need
• k = N/n = the interval size
• randomly select an integer between 1 to k
 • then take every kth unit
All of this will be
much clearer with
an example. Let's
assume that we
have a population
that only has
N=100 people in it
and that you want
to take a sample of
n=20. To use
systematic
sampling, the
population must be
listed in a random
order. The
sampling fraction
would be f =
20/100 = 20%. in
this case, the
interval size, k, is
equal to N/n = 100/20 = 5. Now, select a random integer from 1 to 5. In our example, imagine
that you chose 4. Now, to select the sample, start with the 4th unit in the list and take every k-th
unit (every 5th, because k=5). You would be sampling units 4, 9, 14, 19, and so on to 100 and
you would wind up with 20 units in your sample.
For this to work, it is essential that the units in the population are randomly ordered, at least with
respect to the characteristics you are measuring. Why would you ever want to use systematic
random sampling? For one thing, it is fairly easy to do. You only have to select a single random
number to start things off. It may also be more precise than simple random sampling. Finally, in
some situations there is simply no easier way to do random sampling. For instance, I once had to
do a study that involved sampling from all the books in a library. Once selected, I would have to
go to the shelf, locate the book, and record when it last circulated. I knew that I had a fairly good
sampling frame in the form of the shelf list (which is a card catalog where the entries are
arranged in the order they occur on the shelf). To do a simple random sample, I could have
estimated the total number of books and generated random numbers to draw the sample; but how
would I find book #74,329 easily if that is the number I selected? I couldn't very well count the
cards until I came to 74,329! Stratifying wouldn't solve that problem either. For instance, I could
have stratified by card catalog drawer and drawn a simple random sample within each drawer.
But I'd still be stuck counting cards. Instead, I did a systematic random sample. I estimated the
number of books in the entire collection. Let's imagine it was 100,000. I decided that I wanted to
take a sample of 1000 for a sampling fraction of 1000/100,000 = 1%. To get the sampling
interval k, I divided N/n = 100,000/1000 = 100. Then I selected a random integer between 1 and
100. Let's say I got 57. Next I did a little side study to determine how thick a thousand cards are
in the card catalog (taking into account the varying ages of the cards). Let's say that on average I
found that two cards that were separated by 100 cards were about .75 inches apart in the catalog
drawer. That information gave me everything I needed to draw the sample. I counted to the 57th
by hand and recorded the book information. Then, I took a compass. (Remember those from your
high-school math class? They're the funny little metal instruments with a sharp pin on one end
and a pencil on the other that you used to draw circles in geometry class.) Then I set the compass
at .75", stuck the pin end in at the 57th card and pointed with the pencil end to the next card
(approximately 100 books away). In this way, I approximated selecting the 157th, 257th, 357th,
and so on. I was able to accomplish the entire selection procedure in very little time using this
systematic random sampling approach. I'd probably still be there counting cards if I'd tried
another random sampling method. (Okay, so I have no life. I got compensated nicely, I don't
mind saying, for coming up with this scheme.)
Cluster (Area) Random Sampling
The problem with random sampling methods when we have to sample a population that's
disbursed across a wide geographic region is that you will have to cover a lot of ground
geographically in order to get to each of the units you sampled. Imagine taking a simple random
sample of all the residents of New York State in order to conduct personal interviews. By the
luck of the draw you will wind up with respondents who come from all over the state. Your
interviewers are going to have a lot of traveling to do. It is for precisely this problem that cluster
or area random sampling was invented.
In cluster sampling, we follow these steps:
• divide population into clusters (usually along geographic boundaries)
• randomly sample clusters
• measure all units within sampled clusters
For instance, in the
figure we see a
map of the
counties in New
York State. Let's
say that we have to
do a survey of
town governments
that will require us
going to the towns
personally. If we
do a simple
random sample
state-wide we'll
have to cover the
entire state
geographically.
Instead, we decide
to do a cluster
sampling of five
counties (marked
in red in the
figure). Once these are selected, we go to every town government in the five areas. Clearly this
strategy will help us to economize on our mileage. Cluster or area sampling, then, is useful in
situations like this, and is done primarily for efficiency of administration. Note also, that we
probably don't have to worry about using this approach if we are conducting a mail or telephone
survey because it doesn't matter as much (or cost more or raise inefficiency) where we call or
send letters to.
Multi-Stage Sampling
The four methods we've covered so far -- simple, stratified, systematic and cluster -- are the
simplest random sampling strategies. In most real applied social research, we would use
sampling methods that are considerably more complex than these simple variations. The most
important principle here is that we can combine the simple methods described earlier in a variety
of useful ways that help us address our sampling needs in the most efficient and effective manner
possible. When we combine sampling methods, we call this multi-stage sampling.
For example, consider the idea of sampling New York State residents for face-to-face interviews.
Clearly we would want to do some type of cluster sampling as the first stage of the process. We
might sample townships or census tracts throughout the state. But in cluster sampling we would
then go on to measure everyone in the clusters we select. Even if we are sampling census tracts
we may not be able to measure everyone who is in the census tract. So, we might set up a
stratified sampling process within the clusters. In this case, we would have a two-stage sampling
process with stratified samples within cluster samples. Or, consider the problem of sampling
students in grade schools. We might begin with a national sample of school districts stratified by
economics and educational level. Within selected districts, we might do a simple random sample
of schools. Within schools, we might do a simple random sample of classes or grades. And,
within classes, we might even do a simple random sample of students. In this case, we have three
or four stages in the sampling process and we use both stratified and simple random sampling.
By combining different sampling methods we are able to achieve a rich variety of probabilistic
sampling methods that can be used in a wide range of social research contexts.

Nonprobability Sampling
The difference between nonprobability and probability sampling is that nonprobability sampling
does not involve random selection and probability sampling does. Does that mean that
nonprobability samples aren't representative of the population? Not necessarily. But it does mean
that nonprobability samples cannot depend upon the rationale of probability theory. At least with
a probabilistic sample, we know the odds or probability that we have represented the population
well. We are able to estimate confidence intervals for the statistic. With nonprobability samples,
we may or may not represent the population well, and it will often be hard for us to know how
well we've done so. In general, researchers prefer probabilistic or random sampling methods over
nonprobabilistic ones, and consider them to be more accurate and rigorous. However, in applied
social research there may be circumstances where it is not feasible, practical or theoretically
sensible to do random sampling. Here, we consider a wide range of nonprobabilistic alternatives.
We can divide nonprobability sampling methods into two broad types: accidental or purposive.
Most sampling methods are purposive in nature because we usually approach the sampling
problem with a specific plan in mind. The most important distinctions among these types of
sampling methods are the ones between the different types of purposive sampling approaches.
Accidental, Haphazard or Convenience Sampling
One of the most common methods of sampling goes under the various titles listed here. I would
include in this category the traditional "man on the street" (of course, now it's probably the
"person on the street") interviews conducted frequently by television news programs to get a
quick (although nonrepresentative) reading of public opinion. I would also argue that the typical
use of college students in much psychological research is primarily a matter of convenience.
(You don't really believe that psychologists use college students because they believe they're
representative of the population at large, do you?). In clinical practice,we might use clients who
are available to us as our sample. In many research contexts, we sample simply by asking for
volunteers. Clearly, the problem with all of these types of samples is that we have no evidence
that they are representative of the populations we're interested in generalizing to -- and in many
cases we would clearly suspect that they are not.
Purposive Sampling
In purposive sampling, we sample with a purpose in mind. We usually would have one or more
specific predefined groups we are seeking. For instance, have you ever run into people in a mall
or on the street who are carrying a clipboard and who are stopping various people and asking if
they could interview them? Most likely they are conducting a purposive sample (and most likely
they are engaged in market research). They might be looking for Caucasian females between 30-
40 years old. They size up the people passing by and anyone who looks to be in that category
they stop to ask if they will participate. One of the first things they're likely to do is verify that
the respondent does in fact meet the criteria for being in the sample. Purposive sampling can be
very useful for situations where you need to reach a targeted sample quickly and where sampling
for proportionality is not the primary concern. With a purposive sample, you are likely to get the
opinions of your target population, but you are also likely to overweight subgroups in your
population that are more readily accessible.
All of the methods that follow can be considered subcategories of purposive sampling methods.
We might sample for specific groups or types of people as in modal instance, expert, or quota
sampling. We might sample for diversity as in heterogeneity sampling. Or, we might capitalize
on informal social networks to identify specific respondents who are hard to locate otherwise, as
in snowball sampling. In all of these methods we know what we want -- we are sampling with a
purpose.
• Modal Instance Sampling
In statistics, the mode is the most frequently occurring value in a distribution. In sampling, when
we do a modal instance sample, we are sampling the most frequent case, or the "typical" case. In
a lot of informal public opinion polls, for instance, they interview a "typical" voter. There are a
number of problems with this sampling approach. First, how do we know what the "typical" or
"modal" case is? We could say that the modal voter is a person who is of average age,
educational level, and income in the population. But, it's not clear that using the averages of
these is the fairest (consider the skewed distribution of income, for instance). And, how do you
know that those three variables -- age, education, income -- are the only or even the most
relevant for classifying the typical voter? What if religion or ethnicity is an important
discriminator? Clearly, modal instance sampling is only sensible for informal sampling contexts.
• Expert Sampling
Expert sampling involves the assembling of a sample of persons with known or demonstrable
experience and expertise in some area. Often, we convene such a sample under the auspices of a
"panel of experts." There are actually two reasons you might do expert sampling. First, because it
would be the best way to elicit the views of persons who have specific expertise. In this case,
expert sampling is essentially just a specific subcase of purposive sampling. But the other reason
you might use expert sampling is to provide evidence for the validity of another sampling
approach you've chosen. For instance, let's say you do modal instance sampling and are
concerned that the criteria you used for defining the modal instance are subject to criticism. You
might convene an expert panel consisting of persons with acknowledged experience and insight
into that field or topic and ask them to examine your modal definitions and comment on their
appropriateness and validity. The advantage of doing this is that you aren't out on your own
trying to defend your decisions -- you have some acknowledged experts to back you. The
disadvantage is that even the experts can be, and often are, wrong.
• Quota Sampling
In quota sampling, you select people nonrandomly according to some fixed quota. There are two
types of quota sampling: proportional and non proportional. In proportional quota sampling
you want to represent the major characteristics of the population by sampling a proportional
amount of each. For instance, if you know the population has 40% women and 60% men, and
that you want a total sample size of 100, you will continue sampling until you get those
percentages and then you will stop. So, if you've already got the 40 women for your sample, but
not the sixty men, you will continue to sample men but even if legitimate women respondents
come along, you will not sample them because you have already "met your quota." The problem
here (as in much purposive sampling) is that you have to decide the specific characteristics on
which you will base the quota. Will it be by gender, age, education race, religion, etc.?
Nonproportional quota sampling is a bit less restrictive. In this method, you specify the
minimum number of sampled units you want in each category. here, you're not concerned with
having numbers that match the proportions in the population. Instead, you simply want to have
enough to assure that you will be able to talk about even small groups in the population. This
method is the nonprobabilistic analogue of stratified random sampling in that it is typically used
to assure that smaller groups are adequately represented in your sample.
• Heterogeneity Sampling
We sample for heterogeneity when we want to include all opinions or views, and we aren't
concerned about representing these views proportionately. Another term for this is sampling for
diversity. In many brainstorming or nominal group processes (including concept mapping), we
would use some form of heterogeneity sampling because our primary interest is in getting broad
spectrum of ideas, not identifying the "average" or "modal instance" ones. In effect, what we
would like to be sampling is not people, but ideas. We imagine that there is a universe of all
possible ideas relevant to some topic and that we want to sample this population, not the
population of people who have the ideas. Clearly, in order to get all of the ideas, and especially
the "outlier" or unusual ones, we have to include a broad and diverse range of participants.
Heterogeneity sampling is, in this sense, almost the opposite of modal instance sampling.
• Snowball Sampling
In snowball sampling, you begin by identifying someone who meets the criteria for inclusion in
your study. You then ask them to recommend others who they may know who also meet the
criteria. Although this method would hardly lead to representative samples, there are times when
it may be the best method available. Snowball sampling is especially useful when you are trying
to reach populations that are inaccessible or hard to find. For instance, if you are studying the
homeless, you are not likely to be able to find good lists of homeless people within a specific
geographical area. However, if you go to that area and identify one or two, you may find that
they know very well who the other homeless people in their vicinity are and how you can find
them.
Measurement
Measurement is the process observing and recording the observations that are collected as part of
a research effort. There are two major issues that will be considered here.
First, you have to understand the fundamental ideas involved in measuring. Here we consider
two of major measurement concepts. In Levels of Measurement, I explain the meaning of the
four major levels of measurement: nominal, ordinal, interval and ratio. Then we move on to the
reliability of measurement, including consideration of true score theory and a variety of
reliability estimators.
Second, you have to understand the different types of measures that you might use in social
research. We consider four broad categories of measurements. Survey research includes the
design and implementation of interviews and questionnaires. Scaling involves consideration of
the major methods of developing and implementing a scale. Qualitative research provides an
overview of the broad range of non-numerical measurement approaches. And unobtrusive
measures presents a variety of measurement methods that don't intrude on or interfere with the
context of the research.

Construct Validity
Construct validity refers to the degree to which inferences can legitimately be made from the
operationalizations in your study to the theoretical constructs on which those operationalizations
were based. Like external validity, construct validity is related to generalizing. But, where
external validity involves generalizing from your study context to other people, places or times,
construct validity involves generalizing from your program or measures to the concept of your
program or measures. You might think of construct validity as a "labeling" issue. When you
implement a program that you call a "Head Start" program, is your label an accurate one? When
you measure what you term "self esteem" is that what you were really measuring?
I would like to tell two major stories here. The first is the more straightforward one. I'll discuss
several ways of thinking about the idea of construct validity, several metaphors that might
provide you with a foundation in the richness of this idea. Then, I'll discuss the major construct
validity threats, the kinds of arguments your critics are likely to raise when you make a claim
that your program or measure is valid. In most research methods texts, construct validity is
presented in the section on measurement. And, it is typically presented as one of many different
types of validity (e.g., face validity, predictive validity, concurrent validity) that you might want
to be sure your measures have. I don't see it that way at all. I see construct validity as the
overarching quality with all of the other measurement validity labels falling beneath it. And, I
don't see construct validity as limited only to measurement. As I've already implied, I think it is
as much a part of the independent variable -- the program or treatment -- as it is the dependent
variable. So, I'll try to make some sense of the various measurement validity types and try to
move you to think instead of the validity of any operationalization as falling within the general
category of construct validity, with a variety of subcategories and subtypes.
The second story I want to tell is more historical in nature. During World War II, the U.S.
government involved hundreds (and perhaps thousands) of psychologists and psychology
graduate students in the development of a wide array of measures that were relevant to the war
effort. They needed personality screening tests for prospective fighter pilots, personnel measures
that would enable sensible assignment of people to job skills, psychophysical measures to test
reaction times, and so on. After the war, these psychologists needed to find gainful employment
outside of the military context, and it's not surprising that many of them moved into testing and
measurement in a civilian context. During the early 1950s, the American Psychological
Association began to become increasingly concerned with the quality or validity of all of the new
measures that were being generated and decided to convene an effort to set standards for
psychological measures. The first formal articulation of the idea of construct validity came from
this effort and was couched under the somewhat grandiose idea of the nomological network. The
nomological network provided a theoretical basis for the idea of construct validity, but it didn't
provide practicing researchers with a way to actually establish whether their measures had
construct validity. In 1959, an attempt was made to develop a method for assessing construct
validity using what is called a multitrait-multimethod matrix, or MTMM for short. In order to
argue that your measures had construct validity under the MTMM approach, you had to
demonstrate that there was both convergent and discriminant validity in your measures. You
demonstrated convergent validity when you showed that measures that are theoretically
supposed to be highly interrelated are, in practice, highly interrelated. And, you showed
discriminant validity when you demonstrated that measures that shouldn't be related to each
other in fact were not. While the MTMM did provide a methodology for assessing construct
validity, it was a difficult one to implement well, especially in applied social research contexts
and, in fact, has seldom been formally attempted. When we examine carefully the thinking about
construct validity that underlies both the nomological network and the MTMM, one of the key
themes we can identify in both is the idea of "pattern." When we claim that our programs or
measures have construct validity, we are essentially claiming that we as researchers understand
how our constructs or theories of the programs and measures operate in theory and we claim that
we can provide evidence that they behave in practice the way we think they should. The
researcher essentially has a theory of how the programs and measures related to each other (and
other theoretical terms), a theoretical pattern if you will. And, the researcher provides evidence
through observation that the programs or measures actually behave that way in reality, an
observed pattern. When we claim construct validity, we're essentially claiming that our observed
pattern -- how things operate in reality -- corresponds with our theoretical pattern -- how we
think the world works. I call this process pattern matching, and I believe that it is the heart of
construct validity. It is clearly an underlying theme in both the nomological network and the
MTMM ideas. And, I think that we can develop concrete and feasible methods that enable
practicing researchers to assess pattern matches -- to assess the construct validity of their
research. The section on pattern matching lays out my idea of how we might use this approach to
assess construct validity

Measurement Validity Types

There's an awful lot of confusion in the methodological literature that stems from the wide
variety of labels that are used to describe the validity of measures. I want to make two cases here.
First, it's dumb to limit our scope only to the validity of measures. We really want to talk about
the validity of any operationalization. That is, any time you translate a concept or construct into a
functioning and operating reality (the operationalization), you need to be concerned about how
well you did the translation. This issue is as relevant when we are talking about treatments or
programs as it is when we are talking about measures. (In fact, come to think of it, we could also
think of sampling in this way. The population of interest in your study is the "construct" and the
sample is your operationalization. If we think of it this way, we are essentially talking about the
construct validity of the sampling!). Second, I want to use the term construct validity to refer to
the general case of translating any construct into an operationalization. Let's use all of the other
validity terms to reflect different ways you can demonstrate different aspects of construct
validity.
With all that in mind, here's a list of the validity types that are typically mentioned in texts and
research papers when talking about the quality of measurement:
• Construct validity
○ Translation validity
 Face validity
 Content validity
○ Criterion-related validity
 Predictive validity
 Concurrent validity
 Convergent validity
 Discriminant validity
I have to warn you here that I made this list up. I've never heard of "translation" validity before,
but I needed a good name to summarize what both face and content validity are getting at, and
that one seemed sensible. All of the other labels are commonly known, but the way I've
organized them is different than I've seen elsewhere.
Let's see if we can make some sense out of this list. First, as mentioned above, I would like to
use the term construct validity to be the overarching category. Construct validity is the
approximate truth of the conclusion that your operationalization accurately reflects its construct.
All of the other terms address this general issue in different ways. Second, I make a distinction
between two broad types: translation validity and criterion-related validity. That's because I think
these correspond to the two major ways you can assure/assess the validity of an
operationalization. In translation validity, you focus on whether the operationalization is a good
reflection of the construct. This approach is definitional in nature -- it assumes you have a good
detailed definition of the construct and that you can check the operationalization against it. In
criterion-related validity, you examine whether the operationalization behaves the way it
should given your theory of the construct. This is a more relational approach to construct
validity. it assumes that your operationalization should function in predictable ways in relation to
other operationalizations based upon your theory of the construct. (If all this seems a bit dense,
hang in there until you've gone through the discussion below -- then come back and re-read this
paragraph). Let's go through the specific validity types.
Translation Validity
I just made this one up today! (See how easy it is to be a methodologist?) I needed a term that
described what both face and content validity are getting at. In essence, both of those validity
types are attempting to assess the degree to which you accurately translated your construct into
the operationalization, and hence the choice of name. Let's look at the two types of translation
validity.
Face Validity
In face validity, you look at the operationalization and see whether "on its face" it seems like a
good translation of the construct. This is probably the weakest way to try to demonstrate
construct validity. For instance, you might look at a measure of math ability, read through the
questions, and decide that yep, it seems like this is a good measure of math ability (i.e., the label
"math ability" seems appropriate for this measure). Or, you might observe a teenage pregnancy
prevention program and conclude that, "Yep, this is indeed a teenage pregnancy prevention
program." Of course, if this is all you do to assess face validity, it would clearly be weak
evidence because it is essentially a subjective judgment call. (Note that just because it is weak
evidence doesn't mean that it is wrong. We need to rely on our subjective judgment throughout
the research process. It's just that this form of judgment won't be very convincing to others.) We
can improve the quality of face validity assessment considerably by making it more systematic.
For instance, if you are trying to assess the face validity of a math ability measure, it would be
more convincing if you sent the test to a carefully selected sample of experts on math ability
testing and they all reported back with the judgment that your measure appears to be a good
measure of math ability.
Content Validity
In content validity, you essentially check the operationalization against the relevant content
domain for the construct. This approach assumes that you have a good detailed description of the
content domain, something that's not always true. For instance, we might lay out all of the
criteria that should be met in a program that claims to be a "teenage pregnancy prevention
program." We would probably include in this domain specification the definition of the target
group, criteria for deciding whether the program is preventive in nature (as opposed to treatment-
oriented), and lots of criteria that spell out the content that should be included like basic
information on pregnancy, the use of abstinence, birth control methods, and so on. Then, armed
with these criteria, we could use them as a type of checklist when examining our program. Only
programs that meet the criteria can legitimately be defined as "teenage pregnancy prevention
programs." This all sounds fairly straightforward, and for many operationalizations it will be.
But for other constructs (e.g., self-esteem, intelligence), it will not be easy to decide on the
criteria that constitute the content domain.
Criterion-Related Validity
In criteria-related validity, you check the performance of your operationalization against some
criterion. How is this different from content validity? In content validity, the criteria are the
construct definition itself -- it is a direct comparison. In criterion-related validity, we usually
make a prediction about how the operationalization will perform based on our theory of the
construct. The differences among the different criterion-related validity types is in the criteria
they use as the standard for judgment.
Predictive Validity
In predictive validity, we assess the operationalization's ability to predict something it should
theoretically be able to predict. For instance, we might theorize that a measure of math ability
should be able to predict how well a person will do in an engineering-based profession. We
could give our measure to experienced engineers and see if there is a high correlation between
scores on the measure and their salaries as engineers. A high correlation would provide evidence
for predictive validity -- it would show that our measure can correctly predict something that we
theoretically think it should be able to predict.
Concurrent Validity
In concurrent validity, we assess the operationalization's ability to distinguish between groups
that it should theoretically be able to distinguish between. For example, if we come up with a
way of assessing manic-depression, our measure should be able to distinguish between people
who are diagnosed manic-depression and those diagnosed paranoid schizophrenic. If we want to
assess the concurrent validity of a new measure of empowerment, we might give the measure to
both migrant farm workers and to the farm owners, theorizing that our measure should show that
the farm owners are higher in empowerment. As in any discriminating test, the results are more
powerful if you are able to show that you can discriminate between two groups that are very
similar.
Convergent Validity
In convergent validity, we examine the degree to which the operationalization is similar to
(converges on) other operationalizations that it theoretically should be similar to. For instance, to
show the convergent validity of a Head Start program, we might gather evidence that shows that
the program is similar to other Head Start programs. Or, to show the convergent validity of a test
of arithmetic skills, we might correlate the scores on our test with scores on other tests that
purport to measure basic math ability, where high correlations would be evidence of convergent
validity.
Discriminant Validity
In discriminant validity, we examine the degree to which the operationalization is not similar to
(diverges from) other operationalizations that it theoretically should be not be similar to. For
instance, to show the discriminant validity of a Head Start program, we might gather evidence
that shows that the program is not similar to other early childhood programs that don't label
themselves as Head Start programs. Or, to show the discriminant validity of a test of arithmetic
skills, we might correlate the scores on our test with scores on tests that of verbal ability, where
low correlations would be evidence of discriminant validity.

Idea of Construct Validity

Construct validity refers to the degree to which inferences can legitimately be made from the
operationalizations in your study to the theoretical constructs on which those operationalizations
were based. I find that it helps me to divide the issues into two broad territories that I call the
"land of theory" and the "land of observation." The land of theory is what goes on inside your
mind, and your attempt to explain or articulate this to others. It is all of the ideas, theories,
hunches and hypotheses that you have about the world. In the land of theory you will find your
idea of the program or treatment as it should be. You will find the idea or construct of the
outcomes or measures that you believe you are trying to affect. The land of observation consists
of what you see happening in the world around you and the public manifestations of that world.
In the land of observation you will find your actual program or treatment, and your actual
measures or observational procedures. Presumably, you have constructed the land of observation
based on your theories. You developed the program to reflect the kind of program you had in
mind. You created the measures to get at what you wanted to get at.
Construct validity is an assessment of how well you translated your ideas or theories into actual
programs or measures. Why is this important? Because when you think about the world or talk
about it with others (land of theory) you are using words that represent concepts. If you tell
someone that a special type of math tutoring will help their child do better in math, you are
communicating at the level of concepts or constructs. You aren't describing in operational detail
the specific things that the tutor will do with their child. You aren't describing the specific
questions that will be on the math test that their child will do better on. You are talking in general
terms, using constructs. If you based your recommendation on research that showed that the
special type of tutoring improved children' math scores, you would want to be sure that the type
of tutoring you are referring to is the same as what that study implemented and that the type of
outcome you're saying should occur was the type they measured in their study. Otherwise, you
would be mislabeling or misrepresenting the research. In this sense, construct validity can be
viewed as a "truth in labeling" kind of issue.
There really are two broad ways of looking at the idea of construct validity. I'll call the first the
"definitionalist" perspective because it essentially holds that the way to assure construct validity
is to define the construct so precisely that you can operationalize it in a straightforward manner.
In a definitionalist view, you have either operationalized the construct correctly or you haven't --
it's an either/or type of thinking. Either this program is a "Type A Tutoring Program" or it isn't.
Either you're measuring self esteem or you aren't.
The other perspective I'd call "relationalist." To a relationalist, things are not either/or or black-
and-white -- concepts are more or less related to each other. The meaning of terms or constructs
differs relatively, not absolutely. The program in your study might be a "Type A Tutoring
Program" in some ways, while in others it is not. It might be more that type of program than
another program. Your measure might be capturing a lot of the construct of self esteem, but it
may not capture all of it. There may be another measure that is closer to the construct of self
esteem than yours is. Relationalism suggests that meaning changes gradually. It rejects the idea
that we can rely on operational definitions as the basis for construct definition.
To get a clearer idea of this distinction, you might
think about how the law approaches the construct
of "truth." Most of you have heard the standard
oath that a witness in a U.S. court is expected to
swear. They are to tell "the truth, the whole truth
and nothing but the truth." What does this mean?
If we only had them swear to tell the truth, they
might choose to interpret that as "make sure that
what you say is true." But that wouldn't guarantee
that they would tell everything they knew to be
true. They might leave some important things out.
They would still be telling the truth. They just
wouldn't be telling everything. On the other hand,
they are asked to tell "nothing but the truth." This
suggests that we can say simply that Statement X
is true and Statement Y is not true.
Now, let's see how this oath translates into a measurement and construct validity context. For
instance, we might want our measure to reflect "the construct, the whole construct, and nothing
but the construct." What does this mean? Let's assume that we have five distinct concepts that
are all conceptually related to each other -- self esteem, self worth, self disclosure, self
confidence, and openness. Most people would say that these concepts are similar, although they
can be distinguished from each other. If we were trying to develop a measure of self esteem,
what would it mean to measure "self esteem, all of self esteem, and nothing but self esteem?" If
the concept of self esteem overlaps with the others, how could we possibly measure all of it (that
would presumably include the part that overlaps with others) and nothing but it? We couldn't! If
you believe that meaning is relational in nature -- that some concepts are "closer" in meaning
than others -- then the legal model discussed here does not work well as a model for construct
validity.
In fact, we will see that most social research methodologists have (whether they've thought about
it or not!) rejected the definitionalist perspective in favor of a relationalist one. In order to
establish construct validity you have to meet the following conditions:
• You have to set the construct you want to operationalize (e.g., self esteem) within a
semantic net (or "net of meaning"). This means that you have to tell us what your
construct is more or less similar to in meaning.
• You need to be able to provide direct evidence that you control the operationalization of
the construct -- that your operationalizations look like what they should theoretically look
like. If you are trying to measure self esteem, you have to be able to explain why you
operationalized the questions the way you did. If all of your questions are addition
problems, how can you argue that your measure reflects self esteem and not adding
ability?
• You have to provide evidence that your data support your theoretical view of the
relations among constructs. If you believe that self esteem is closer in meaning to self
worth than it is to anxiety, you should be able to show that measures of self esteem are
more highly correlated with measures of self worth than with ones of anxiety.
• Convergent & Discriminant Validity
• Convergent and discriminant validity are both considered subcategories or subtypes of
construct validity. The important thing to recognize is that they work together -- if you
can demonstrate that you have evidence for both convergent and discriminant validity,
then you've by definition demonstrated that you have evidence for construct validity. But,
neither one alone is sufficient for establishing construct validity.
• I find it easiest to think about convergent and discriminant validity as two inter-locking
propositions. In simple words I would describe what they are doing as follows:
measures of constructs that theoretically should be related to each other
are, in fact, observed to be related to each other (that is, you should be
able to show a correspondence or convergence between similar
constructs)
and
measures of constructs that theoretically should not be related to each
other are, in fact, observed to not be related to each other (that is, you
should be able to discriminate between dissimilar constructs)
• To estimate the degree to which any two measures are related to each other we typically
use the correlation coefficient. That is, we look at the patterns of intercorrelations among
our measures. Correlations between theoretically similar measures should be "high" while
correlations between theoretically dissimilar measures should be "low".
• The main problem that I have with this convergent-discrimination idea has to do with my
use of the quotations around the terms "high" and "low" in the sentence above. The
question is simple -- how "high" do correlations need to be to provide evidence for
convergence and how "low" do they need to be to provide evidence for discrimination?
And the answer is -- we don't know! In general we want convergent correlations to be as
high as possible and discriminant ones to be as low as possible, but there is no hard and
fast rule. Well, let's not let that stop us. One thing that we can say is that the convergent
correlations should always be higher than the discriminant ones. At least that helps a bit.
• Before we get too deep into the idea of convergence and discrimination, let's take a look
at each one using a simple example.
• Convergent Validity
• To establish convergent validity, you need to show that measures that should be related
are in reality related. In the figure below, we see four measures (each is an item on a
scale) that all purport to reflect the construct of self esteem. For instance, Item 1 might be
the statement "I feel good about myself" rated using a 1-to-5 Likert-type response format.
We theorize that all four items reflect the idea of self esteem (this is why I labeled the top
part of the figure Theory). On the bottom part of the figure (Observation) we see the
intercorrelations of the four scale items. This might be based on giving our scale out to a
sample of respondents. You should readily see that the item intercorrelations for all item
pairings are very high (remember that correlations range from -1.00 to +1.00). This
provides evidence that our theory that all four items are related to the same construct is
supported.
 •

• Notice, however, that while the high intercorrelations demonstrate the the four items are
probably related to the same construct, that doesn't automatically mean that the construct
is self esteem. Maybe there's some other construct that all four items are related to (more
about this later). But, at the very least, we can assume from the pattern of correlations
that the four items are converging on the same thing, whatever we might call it.
• Discriminant Validity
• To establish discriminant validity, you need to show that measures that should not be
related are in reality not related. In the figure below, we again see four measures (each is
an item on a scale). Here, however, two of the items are thought to reflect the construct of
self esteem while the other two are thought to reflect locus of control. The top part of the
figure shows our theoretically expected relationships among the four items. If we have
discriminant validity, the relationship between measures from different constructs should
be very low (again, we don't know how low "low" should be, but we'll deal with that
later). There are four correlations between measures that reflect different constructs, and
these are shown on the bottom of the figure (Observation). You should see immediately
that these four cross-construct correlations are very low (i.e., near zero) and certainly
much lower than the convergent correlations in the previous figure.
 •

• As above, just because we've provided evidence that the two sets of two measures each
seem to be related to different constructs (because their intercorrelations are so low)
doesn't mean that the constructs they're related to are self esteem and locus of control.
But the correlations do provide evidence that the two sets of measures are discriminated
from each other.
• Putting It All Together
• OK, so where does this leave us? I've shown how we go about providing evidence for
convergent and discriminant validity separately. But as I said at the outset, in order to
argue for construct validity we really need to be able to show that both of these types of
validity are supported. Given the above, you should be able to see that we could put both
principles together into a single analysis to examine both at the same time. This is
illustrated in the figure below.
• The figure shows six measures, three that are theoretically related to the construct of self
esteem and three that are thought to be related to locus of control. The top part of the
figure shows this theoretical arrangement. The bottom of the figure shows what a
correlation matrix based on a pilot sample might show. To understand this table, you
need to first be able to identify the convergent correlations and the discriminant ones.
There are two sets or blocks of convergent coefficients (in green), one 3x3 block for the
self esteem intercorrelations and one 3x3 block for the locus of control correlations.
There are also two 3x3 blocks of discriminant coefficients (shown in red), although if
you're really sharp you'll recognize that they are the same values in mirror image (Do you
know why? You might want to read up on correlations to refresh your memory).
• How do we make sense of the patterns of correlations? Remember that I said above that
we don't have any firm rules for how high or low the correlations need to be to provide
evidence for either type of validity. But we do know that the convergent correlations
should always be higher than the discriminant ones. take a good look at the table and you
will see that in this example the convergent correlations are always higher than the
discriminant ones. I would conclude from this that the correlation matrix provides
evidence for both convergent and discriminant validity, all in one analysis!
 •

• But while the pattern supports discriminant and convergent validity, does it show that the
three self esteem measures actually measure self esteem or that the three locus of control
measures actually measure locus of control. Of course not. That would be much too easy.
• So, what good is this analysis? It does show that, as you predicted, the three self esteem
measures seem to reflect the same construct (whatever that might be), the three locus of
control measures also seem to reflect the same construct (again, whatever that is) and that
the two sets of measures seem to be reflecting two different constructs (whatever they
are). That's not bad for one simple analysis.
• OK, so how do we get to the really interesting question? How do we show that our
measures are actually measuring self esteem or locus of control? I hate to disappoint you,
but there is no simple answer to that (I bet you knew that was coming). There's a number
of things we can do to address that question. First, we can use other ways to address
construct validity to help provide further evidence that we're measuring what we say
we're measuring. For instance, we might use a face validity or content validity approach
to demonstrate that the measures reflect the constructs we say they are (see the discussion
on types of construct validity for more information).
• One of the most powerful approaches is to include even more constructs and measures.
The more complex our theoretical model (if we find confirmation of the correct pattern in
the correlations), the more we are providing evidence that we know what we're talking
about (theoretically speaking). Of course, it's also harder to get all the correlations to give
you the exact right pattern as you add lots more measures. And, in many studies we
simply don't have the luxury to go adding more and more measures because it's too costly
or demanding. Despite the impracticality, if we can afford to do it, adding more
constructs and measures will enhance our ability to assess construct validity using
approaches like the multitrait-multimethod matrix and the nomological network.
 • Perhaps the most interesting approach to getting at construct validity involves the idea of
pattern matching. Instead of viewing convergent and discriminant validity as differences
of kind, pattern matching views them as differences in degree. This seems a more
reasonable idea, and helps us avoid the problem of how high or low correlations need to
be to say that we've established convergence or discrimination.

Threats to Construct Validity

Before we launch into a discussion of the most common threats to construct validity, let's recall
what a threat to validity is. In a research study you are likely to reach a conclusion that your
program was a good operationalization of what you wanted and that your measures reflected
what you wanted them to reflect. Would you be correct? How will you be criticized if you make
these types of claims? How might you strengthen your claims. The kinds of questions and issues
your critics will raise are what I mean by threats to construct validity.
I take the list of threats from the discussion in Cook and Campbell (Cook, T.D. and Campbell,
D.T. Quasi-Experimentation: Design and Analysis Issues for Field Settings. Houghton Mifflin,
Boston, 1979). While I love their discussion, I do find some of their terminology less than
straightforward -- a lot of what I'll do here is try to explain this stuff in terms that the rest of us
might hope to understand.
Inadequate Preoperational Explication of Constructs
This one isn't nearly as ponderous as it sounds. Here, preoperational means before translating
constructs into measures or treatments, and explication means explanation -- in other words,
you didn't do a good enough job of defining (operationally) what you mean by the construct.
How is this a threat? Imagine that your program consisted of a new type of approach to
rehabilitation. Your critic comes along and claims that, in fact, your program is neither new nor a
true rehabilitation program. You are being accused of doing a poor job of thinking through your
constructs. Some possible solutions:
• think through your concepts better
• use methods (e.g., concept mapping) to articulate your concepts
• get experts to critique your operationalizations
Mono-Operation Bias
Mono-operation bias pertains to the independent variable, cause, program or treatment in your
study -- it does not pertain to measures or outcomes (see Mono-method Bias below). If you only
use a single version of a program in a single place at a single point in time, you may not be
capturing the full breadth of the concept of the program. Every operationalization is flawed
relative to the construct on which it is based. If you conclude that your program reflects the
construct of the program, your critics are likely to argue that the results of your study only reflect
the peculiar version of the program that you implemented, and not the actual construct you had
in mind. Solution: try to implement multiple versions of your program.
Mono-Method Bias
Mono-method bias refers to your measures or observations, not to your programs or causes.
Otherwise, it's essentially the same issue as mono-operation bias. With only a single version of a
self esteem measure, you can't provide much evidence that you're really measuring self esteem.
Your critics will suggest that you aren't measuring self esteem -- that you're only measuring part
of it, for instance. Solution: try to implement multiple measures of key constructs and try to
demonstrate (perhaps through a pilot or side study) that the measures you use behave as you
theoretically expect them to.
Interaction of Different Treatments
You give a new program designed to encourage high-risk teenage girls to go to school and not
become pregnant. The results of your study show that the girls in your treatment group have
higher school attendance and lower birth rates. You're feeling pretty good about your program
until your critics point out that the targeted at-risk treatment group in your study is also likely to
be involved simultaneously in several other programs designed to have similar effects. Can you
really label the program effect as a consequence of your program? The "real" program that the
girls received may actually be the combination of the separate programs they participated in.
Interaction of Testing and Treatment
Does testing or measurement itself make the groups more sensitive or receptive to the treatment?
If it does, then the testing is in effect a part of the treatment, it's inseparable from the effect of the
treatment. This is a labeling issue (and, hence, a concern of construct validity) because you want
to use the label "program" to refer to the program alone, but in fact it includes the testing.
Restricted Generalizability Across Constructs
This is what I like to refer to as the "unintended consequences" treat to construct validity. You do
a study and conclude that Treatment X is effective. In fact, Treatment X does cause a reduction
in symptoms, but what you failed to anticipate was the drastic negative consequences of the side
effects of the treatment. When you say that Treatment X is effective, you have defined
"effective" as only the directly targeted symptom. This threat reminds us that we have to be
careful about whether our observed effects (Treatment X is effective) would generalize to other
potential outcomes.
Confounding Constructs and Levels of Constructs
Imagine a study to test the effect of a new drug treatment for cancer. A fixed dose of the drug is
given to a randomly assigned treatment group and a placebo to the other group. No treatment
effects are detected. Perhaps the result that's observed is only true for that dosage level. Slight
increases or decreases of the dosage may radically change the results. In this context, it is not
"fair" for you to use the label for the drug as a description for your treatment because you only
looked at a narrow range of dose. Like the other construct validity threats, this is essentially a
labeling issue -- your label is not a good description for what you implemented.
The "Social" Threats to Construct Validity
I've set aside the other major threats to construct validity because they all stem from the social
and human nature of the research endeavor.
Hypothesis Guessing
Most people don't just participate passively in a research project. They are trying to figure out
what the study is about. They are "guessing" at what the real purpose of the study is. And, they
are likely to base their behavior on what they guess, not just on your treatment. In an educational
study conducted in a classroom, students might guess that the key dependent variable has to do
with class participation levels. If they increase their participation not because of your program
but because they think that's what you're studying, then you cannot label the outcome as an effect
of the program. It is this labeling issue that makes this a construct validity threat.
Evaluation Apprehension
Many people are anxious about being evaluated. Some are even phobic about testing and
measurement situations. If their apprehension makes them perform poorly (and not your program
conditions) then you certainly can't label that as a treatment effect. Another form of evaluation
apprehension concerns the human tendency to want to "look good" or "look smart" and so on. If,
in their desire to look good, participants perform better (and not as a result of your program!)
then you would be wrong to label this as a treatment effect. In both cases, the apprehension
becomes confounded with the treatment itself and you have to be careful about how you label the
outcomes.
Experimenter Expectancies
These days, where we engage in lots of non-laboratory applied social research, we generally
don't use the term "experimenter" to describe the person in charge of the research. So, let's
relabel this threat "researcher expectancies." The researcher can bias the results of a study in
countless ways, both consciously or unconsciously. Sometimes the researcher can communicate
what the desired outcome for a study might be (and participant desire to "look good" leads them
to react that way). For instance, the researcher might look pleased when participants give a
desired answer. If this is what causes the response, it would be wrong to label the response as a
treatment effect.

The Nomological Network

What is the Nomological Net?
The nomological network is an idea that was developed by Lee Cronbach and Paul Meehl in
1955 (Cronbach, L. and Meehl, P. (1955). Construct validity in psychological tests,
Psychological Bulletin, 52, 4, 281-302.) as part of the American Psychological Association's
efforts to develop standards for psychological testing. The term "nomological" is derived from
Greek and means "lawful", so the nomological network can be thought of as the "lawful
network." The nomological network was Cronbach and Meehl's view of construct validity. That
is, in order to provide evidence that your measure has construct validity, Cronbach and Meehl
argued that you had to develop a nomological network for your measure. This network would
include the theoretical framework for what you are trying to measure, an empirical framework
for how you are going to measure it, and specification of the linkages among and between these
two frameworks.
The nomological network is founded on a number of principles that guide the researcher when
trying to establish construct validity. They are:
• Scientifically, to make clear what something is or means, so that laws can be set forth in
which that something occurs.
• The laws in a nomological network may relate:
○ observable properties or quantities to each other
○ different theoretical constructs to each other
○ theoretical constructs to observables
• At least some of the laws in the network must involve observables.
• "Learning more about" a theoretical construct is a matter of elaborating the nomological
network in which it occurs or of increasing the definiteness of its components.
• The basic rule for adding a new construct or relation to a theory is that it must generate
laws (nomologicals) confirmed by observation or reduce the number of nomologicals
required to predict some observables.
• Operations which are qualitatively different "overlap" or "measure the same thing" if
their positions in the nomological net tie them to the same construct variable.
What Cronbach and Meehl were trying to do is to link the conceptual/theoretical realm with the
observable one, because this is the central concern of construct validity. While the nomological
network idea may work as a philosophical foundation for construct validity, it does not provide a
practical and usable methodology for actually assessing construct validity. The next phase in the
evolution of the idea of construct validity -- the development of the multitrait-multimethod
matrix -- moved us a bit further toward a methodological approach to construct validity.
What is the Multitrait-Multimethod Matrix?
The Multitrait-Multimethod Matrix (hereafter labeled MTMM) is an approach to assessing the
construct validity of a set of measures in a study. It was developed in 1959 by Campbell and
Fiske (Campbell, D. and Fiske, D. (1959). Convergent and discriminant validation by the
multitrait-multimethod matrix. 56, 2, 81-105.) in part as an attempt to provide a practical
methodology that researchers could actually use (as opposed to the nomological network idea
which was theoretically useful but did not include a methodology). Along with the MTMM,
Campbell and Fiske introduced two new types of validity -- convergent and discriminant -- as
subcategories of construct validity. Convergent validity is the degree to which concepts that
should be related theoretically are interrelated in reality. Discriminant validity is the degree to
which concepts that should not be related theoretically are, in fact, not interrelated in reality. You
can assess both convergent and discriminant validity using the MTMM. In order to be able to
claim that your measures have construct validity, you have to demonstrate both convergence and
discrimination.

The MTMM is simply a matrix or table of correlations arranged to facilitate the interpretation of
the assessment of construct validity. The MTMM assumes that you measure each of several
concepts (called traits by Campbell and Fiske) by each of several methods (e.g., a paper-and-
pencil test, a direct observation, a performance measure). The MTMM is a very restrictive
methodology -- ideally you should measure each concept by each method.
To construct an MTMM, you need to arrange the correlation matrix by concepts within methods.
The figure shows an MTMM for three concepts (traits A, B and C) each of which is measured
with three different methods (1, 2 and 3) Note that you lay the matrix out in blocks by method.
Essentially, the MTMM is just a correlation matrix between your measures, with one exception
-- instead of 1's along the diagonal (as in the typical correlation matrix) we substitute an estimate
of the reliability of each measure as the diagonal.
Before you can interpret an MTMM, you have to understand how to identify the different parts
of the matrix. First, you should note that the matrix is consists of nothing but correlations. It is a
square, symmetric matrix, so we only need to look at half of it (the figure shows the lower
triangle). Second, these correlations can be grouped into three kinds of shapes: diagonals,
triangles, and blocks. The specific shapes are:
• The Reliability Diagonal
(monotrait-monomethod)
Estimates of the reliability of each measure in the matrix. You can estimate reliabilities a number
of different ways (e.g., test-retest, internal consistency). There are as many correlations in the
reliability diagonal as there are measures -- in this example there are nine measures and nine
reliabilities. The first reliability in the example is the correlation of Trait A, Method 1 with Trait
A, Method 1 (hereafter, I'll abbreviate this relationship A1-A1). Notice that this is essentially the
correlation of the measure with itself. In fact such a correlation would always be perfect (i.e.,
r=1.0). Instead, we substitute an estimate of reliability. You could also consider these values to
be monotrait-monomethod correlations.
• The Validity Diagonals
(monotrait-heteromethod)
Correlations between measures of the same trait measured using different methods. Since the
MTMM is organized into method blocks, there is one validity diagonal in each method block.
For example, look at the A1-A2 correlation of .57. This is the correlation between two measures
of the same trait (A) measured with two different measures (1 and 2). Because the two measures
are of the same trait or concept, we would expect them to be strongly correlated. You could also
consider these values to be monotrait-heteromethod correlations.
• The Heterotrait-Monomethod Triangles
These are the correlations among measures that share the same method of measurement. For
instance, A1-B1 = .51 in the upper left heterotrait-monomethod triangle. Note that what these
correlations share is method, not trait or concept. If these correlations are high, it is because
measuring different things with the same method results in correlated measures. Or, in more
straightforward terms, you've got a strong "methods" factor.
• Heterotrait-Heteromethod Triangles
These are correlations that differ in both trait and method. For instance, A1-B2 is .22 in the
example. Generally, because these correlations share neither trait nor method we expect them to
be the lowest in the matrix.
• The Monomethod Blocks
These consist of all of the correlations that share the same method of measurement. There are as
many blocks as there are methods of measurement.
• The Heteromethod Blocks
These consist of all correlations that do not share the same methods. There are (K(K-1))/2 such
blocks, where K = the number of methods. In the example, there are 3 methods and so there are
(3(3-1))/2 = (3(2))/2 = 6/2 = 3 such blocks.
Principles of Interpretation
Now that you can identify the different parts of the MTMM, you can begin to understand the
rules for interpreting it. You should realize that MTMM interpretation requires the researcher to
use judgment. Even though some of the principles may be violated in an MTMM, you may still
wind up concluding that you have fairly strong construct validity. In other words, you won't
necessarily get perfect adherence to these principles in applied research settings, even when you
do have evidence to support construct validity. To me, interpreting an MTMM is a lot like a
physician's reading of an x-ray. A practiced eye can often spot things that the neophyte misses! A
researcher who is experienced with MTMM can use it identify weaknesses in measurement as
well as for assessing construct validity.

To help make the principles more concrete, let's make the example a bit more realistic. We'll
imagine that we are going to conduct a study of sixth grade students and that we want to measure
three traits or concepts: Self Esteem (SE), Self Disclosure (SD) and Locus of Control (LC).
Furthermore, let's measure each of these three different ways: a Paper-and-Pencil (P&P)
measure, a Teacher rating, and a Parent rating. The results are arrayed in the MTMM. As the
principles are presented, try to identify the appropriate coefficients in the MTMM and make a
judgement yourself about the strength of construct validity claims.
The basic principles or rules for the MTMM are:
• Coefficients in the reliability diagonal should consistently be the highest in the matrix.
That is, a trait should be more highly correlated with itself than with anything else! This is
uniformly true in our example.
• Coefficients in the validity diagonals should be significantly different from zero and high
enough to warrant further investigation.
This is essentially evidence of convergent validity. All of the correlations in our example meet
this criterion.
• A validity coefficient should be higher than values lying in its column and row in the
same heteromethod block.
In other words, (SE P&P)-(SE Teacher) should be greater than (SE P&P)-(SD Teacher), (SE
P&P)-(LC Teacher), (SE Teacher)-(SD P&P) and (SE Teacher)-(LC P&P). This is true in all
cases in our example.
• A validity coefficient should be higher than all coefficients in the heterotrait-
monomethod triangles.
This essentially emphasizes that trait factors should be stronger than methods factors. Note that
this is not true in all cases in our example. For instance, the (LC P&P)-(LC Teacher) correlation
of .46 is less than (SE Teacher)-(SD Teacher), (SE Teacher)-(LC Teacher), and (SD Teacher)-
(LC Teacher) -- evidence that there might me a methods factor, especially on the Teacher
observation method.
• The same pattern of trait interrelationship should be seen in all triangles.
The example clearly meets this criterion. Notice that in all triangles the SE-SD relationship is
approximately twice as large as the relationships that involve LC.
Advantages and Disadvantages of MTMM
The MTMM idea provided an operational methodology for assessing construct validity. In the
one matrix it was possible to examine both convergent and discriminant validity simultaneously.
By its inclusion of methods on an equal footing with traits, Campbell and Fiske stressed the
importance of looking for the effects of how we measure in addition to what we measure. And,
MTMM provided a rigorous framework for assessing construct validity.
Despite these advantages, MTMM has received little use since its introduction in 1959. There are
several reasons. First, in its purest form, MTMM requires that you have a fully-crossed
measurement design -- each of several traits is measured by each of several methods. While
Campbell and Fiske explicitly recognized that one could have an incomplete design, they
stressed the importance of multiple replication of the same trait across method. In some applied
research contexts, it just isn't possible to measure all traits with all desired methods (would you
use an "observation" of weight?). In most applied social research, it just wasn't feasible to make
methods an explicit part of the research design. Second, the judgmental nature of the MTMM
may have worked against its wider adoption (although it should actually be perceived as a
strength). many researchers wanted a test for construct validity that would result in a single
statistical coefficient that could be tested -- the equivalent of a reliability coefficient. It was
impossible with MTMM to quantify the degree of construct validity in a study. Finally, the
judgmental nature of MTMM meant that different researchers could legitimately arrive at
different conclusions.
A Modified MTMM -- Leaving out the Methods Factor
As mentioned above, one of the
most difficult aspects of MTMM
from an implementation point of
view is that it required a design
that included all combinations of
both traits and methods. But the
ideas of convergent and
discriminant validity do not
require the methods factor. To see
this, we have to reconsider what Campbell and Fiske meant by convergent and discriminant
validity.
What is convergent validity?
It is the principle that measures of
theoretically similar constructs
should be highly intercorrelated.
We can extend this idea further by
thinking of a measure that has
multiple items, for instance, a
four-item scale designed to
measure self-esteem. If each of the
items actually does reflect the
construct of self-esteem, then we
would expect the items to be
highly intercorrelated as shown in
the figure. These strong
intercorrelations are evidence in
support of convergent validity.
And what is
discriminant
validity?
It is the principle that measures of theoretically different constructs should not correlate highly
with each other. We can see that in the example that shows two constructs -- self-esteem and
locus of control -- each measured in two instruments. We would expect that, because these are
measures of different constructs, the cross-construct correlations would be low, as shown in the
figure. These low correlations are evidence for validity. Finally, we can put this all together to
see how we can address both convergent and discriminant validity simultaneously. Here, we
have two constructs -- self-esteem and locus of control -- each measured with three instruments.
The red and green correlations are within-construct ones. They are a reflection of convergent
validity and should be strong. The blue correlations are cross-construct and reflect discriminant
validity. They should be uniformly lower than the convergent coefficients.
The important thing to notice about this matrix is that it does not explicitly include a methods
factor as a true MTMM would. The matrix examines both convergent and discriminant validity
(like the MTMM) but it only explicitly looks at construct intra- and interrelationships. We can
see in this example that the MTMM idea really had two major themes. The first was the idea of
looking simultaneously at the pattern of convergence and discrimination. This idea is similar in
purpose to the notions implicit in the nomological network -- we are looking at the pattern of
interrelationships based upon our theory of the nomological net. The second idea in MTMM was
the emphasis on methods as a potential confounding factor.
While methods may confound the results, they won't necessarily do so in any given study. And,
while we need to examine our results for the potential for methods factors, it may be that
combining this desire to assess the confound with the need to assess construct validity is more
than one methodology can feasibly handle. Perhaps if we split the two agendas, we will find that
the possibility that we can examine convergent and discriminant validity is greater. But what do
we do about methods factors? One way to deal with them is through replication of research
projects, rather than trying to incorporate a methods test into a single research study. Thus, if we
find a particular outcome in a study using several measures, we might see if that same outcome is
obtained when we replicate the study using different measures and methods of measurement for
the same constructs. The methods issue is considered more as an issue of generalizability (across
measurement methods) rather than one of construct validity.
When viewed this way, we have moved from the idea of a MTMM to that of the multitrait matrix
that enables us to examine convergent and discriminant validity, and hence construct validity.
We will see that when we move away from the explicit consideration of methods and when we
begin to see convergence and discrimination as differences of degree, we essentially have the
foundation for the pattern matching approach to assessing construct validity.

Pattern Matching for Construct Validity

The idea of using pattern matching as a rubric for assessing construct validity is an area where I
have tried to make a contribution (Trochim, W., (1985). Pattern matching, validity, and
conceptualization in program evaluation. Evaluation Review, 9, 5, 575-604 and Trochim, W.
(1989). Outcome pattern matching and program theory. Evaluation and Program Planning, 12,
355-366.), although my work was very clearly foreshadowed, especially in much of Donald T.
Campbell's writings. Here, I'll try to explain what I mean by pattern matching with respect to
construct validity.
The Theory of Pattern Matching
A pattern is any arrangement of objects or entities. The term "arrangement" is used here to
indicate that a pattern is by definition non-random and at least potentially describable. All
theories imply some pattern, but theories and patterns are not the same thing. In general, a theory
postulates structural relationships between key constructs. The theory can be used as the basis for
generating patterns of predictions. For instance, E=MC2 can be considered a theoretical
formulation. A pattern of expectations can be developed from this formula by generating
predicted values for one of these variables given fixed values of the others. Not all theories are
stated in mathematical form, especially in applied social research, but all theories provide
information that enables the generation of patterns of predictions.
Pattern matching
always involves an
attempt to link two
patterns where one
is a theoretical
pattern and the
other is an
observed or
operational one.
The top part of the
figure shows the
realm of theory.
The theory might
originate from a
formal tradition of
theorizing, might
be the ideas or
"hunches" of the
investigator, or
might arise from
some combination
of these. The
conceptualization
task involves the
translation of these
ideas into a specifiable theoretical pattern indicated by the top shape in the figure. The bottom
part of the figure indicates the realm of observation. This is broadly meant to include direct
observation in the form of impressions, field notes, and the like, as well as more formal objective
measures. The collection or organization of relevant operationalizations (i.e., relevant to the
theoretical pattern) is termed the observational pattern and is indicated by the lower shape in the
figure. The inferential task involves the attempt to relate, link or match these two patterns as
indicated by the double arrow in the center of the figure. To the extent that the patterns match,
one can conclude that the theory and any other theories which might predict the same observed
pattern receive support.
It is important to demonstrate that there are no plausible alternative theories that account for the
observed pattern and this task is made much easier when the theoretical pattern of interest is a
unique one. In effect, a more complex theoretical pattern is like a unique fingerprint which one is
seeking in the observed pattern. With more complex theoretical patterns it is usually more
difficult to construe sensible alternative patterns that would also predict the same result. To the
extent that theoretical and observed patterns do not match, the theory may be incorrect or poorly
formulated, the observations may be inappropriate or inaccurate, or some combination of both
states may exist.
All research employs pattern matching principles, although this is seldom done consciously. In
the traditional two-group experimental context, for instance, the typical theoretical outcome
pattern is the hypothesis that there will be a significant difference between treated and untreated
groups. The observed outcome pattern might consist of the averages for the two groups on one or
more measures. The pattern match is accomplished by a test of significance such as the t-test or
ANOVA. In survey research, pattern matching forms the basis of generalizations across different
concepts or population subgroups. In qualitative research pattern matching lies at the heart of any
attempt to conduct thematic analyses.
While current research methods can be described in pattern matching terms, the idea of pattern
matching implies more, and suggests how one might improve on these current methods.
Specifically, pattern matching implies that more complex patterns, if matched, yield greater
validity for the theory. Pattern matching does not differ fundamentally from traditional
hypothesis testing and model building approaches. A theoretical pattern is a hypothesis about
what is expected in the data. The observed pattern consists of the data that are used to examine
the theoretical model. The major differences between pattern matching and more traditional
hypothesis testing approaches are that pattern matching encourages the use of more complex or
detailed hypotheses and treats the observations from a multivariate rather than a univariate
perspective.
Pattern Matching and Construct Validity
While pattern matching can be used to address a variety of questions in social research, the
emphasis here is on its use in assessing construct validity.
The accompanying figure shows the pattern matching structure for an example involving five
measurement constructs -- arithmetic, algebra, geometry, spelling, and reading. In this example,
we'll use concept mapping to develop the theoretical pattern among these constructs. In the
concept mapping we generate a large set of potential arithmetic, algebra, geometry, spelling, and
reading questions. We sort them into piles of similar questions and develop a map that shows
each question in relation to the others. On the map, questions that are more similar are closer to
each other, those less similar are more distant. From the map, we can find the straight-line
distances between all pair of points (i.e., all questions). This is the matrix of interpoint distances.
We might use the questions from the map in constructing our measurement instrument, or we
might sample from these questions. On the observed side, we have one or more test instruments
that contain a number of questions about arithmetic, algebra, geometry, spelling, and reading.
We analyze the data and construct a matrix of inter-item correlations.
What we want to do is compare the matrix of interpoint distances from our concept map (i.e., the
theoretical pattern) with the correlation matrix of the questions (i.e., the observed pattern). How
do we achieve this? Let's assume that we had 100 prospective questions on our concept map, 20
for each construct. Correspondingly, we have 100 questions on our measurement instrument, 20
in each area. Thus, both matrices are 100x100 in size. Because both matrices are symmetric, we
actually have (N(N-1))/2 = (100(99))/2 = 9900/2 = 4,950 unique pairs (excluding the diagonal).
If we "string out" the values in each matrix we can construct a vector or column of 4,950
numbers for each matrix. The first number is the value comparing pair (1,2), the next is (1,3) and
so on to (N-1, N) or (99, 100). Now, we can compute the overall correlation between these two
columns, which is the correlation between our theoretical and observed patterns, the "pattern
matching correlation." In this example, let's assume it is -.93. Why would it be a negative
correlation? Because we are correlating distances on the map with the similarities in the
correlations and we expect that greater distance on the map should be associated with lower
correlation and less distance with greater correlation.
The pattern matching correlation is our overall estimate of the degree of construct validity in this
example because it estimates the degree to which the operational measures reflect our theoretical
expectations.

Advantages and Disadvantages of Pattern Matching

There are several disadvantages of the pattern matching approach to construct validity. The most
obvious is that pattern matching requires that you specify your theory of the constructs rather
precisely. This is typically not done in applied social research, at least not to the level of
specificity implied here. But perhaps it should be done. Perhaps the more restrictive assumption
is that you are able to structure the theoretical and observed patterns the same way so that you
can directly correlate them. We needed to quantify both patterns and, ultimately, describe them
in matrices that had the same dimensions. In most research as it is currently done it will be
relatively easy to construct a matrix of the inter-item correlations. But we seldom currently use
methods like concept mapping that enable us to estimate theoretical patterns that can be linked
with observed ones. Again, perhaps we ought to do this more frequently.
There are a number of advantages of the pattern matching approach, especially relative to the
multitrait-multimethod matrix (MTMM). First, it is more general and flexible than MTMM. It
does not require that you measure each construct with multiple methods. Second, it treats
convergence and discrimination as a continuum. Concepts are more or less similar and so their
interrelations would be more or less convergent or discriminant. This moves the
convergent/discriminant distinction away from the simplistic dichotomous categorical notion to
one that is more suitably post-positivist and continuous in nature. Third, the pattern matching
approach does make it possible to estimate the overall construct validity for a set of measures in
a specific context. Notice that we don't estimate construct validity for a single measure. That's
because construct validity, like discrimination, is always a relative metric. Just as we can only
ask whether you have distinguished something if there is something to distinguish it from, we
can only assess construct validity in terms of a theoretical semantic or nomological net, the
conceptual context within which it resides. The pattern matching correlation tells us, for our
particular study, whether there is a demonstrable relationship between how we theoretically
expect our measures will interrelate and how they do in practice. Finally, because pattern
matching requires a more specific theoretical pattern than we typically articulate, it requires us to
specify what we think about the constructs in our studies. Social research has long been criticized
for conceptual sloppiness, for re-packaging old constructs in new terminology and failing to
develop an evolution of research around key theoretical constructs. Perhaps the emphasis on
theory articulation in pattern matching would encourage us to be more careful about the
conceptual underpinnings of our empirical work. And, after all, isn't that what construct validity
is all about?

Reliability
Reliability has to do with the quality of measurement. In its everyday sense, reliability is the
"consistency" or "repeatability" of your measures. Before we can define reliability precisely we
have to lay the groundwork. First, you have to learn about the foundation of reliability, the true
score theory of measurement. Along with that, you need to understand the different types of
measurement error because errors in measures play a key role in degrading reliability. With this
foundation, you can consider the basic theory of reliability, including a precise definition of
reliability. There you will find out that we cannot calculate reliability -- we can only estimate it.
Because of this, there a variety of different types of reliability that each have multiple ways to
estimate reliability for that type. In the end, it's important to integrate the idea of reliability with
the other major criteria for the quality of measurement -- validity -- and develop an
understanding of the relationships between reliability and validity in measurement.

True Score
Theory
True Score Theory is a theory
about measurement. Like all
theories, you need to recognize
that it is not proven -- it is
postulated as a model of how
the world operates. Like many
very powerful model, the true
score theory is a very simple
one. Essentially, true score
theory maintains that every
measurement is an additive composite of two components: true ability (or the true level) of the
respondent on that measure; and random error. We observe the measurement -- the score on the
test, the total for a self-esteem instrument, the scale value for a person's weight. We don't observe
what's on the right side of the equation (only God knows what those values are!), we assume that
there are two components to the right side.
The simple equation of X = T + eX has a parallel equation at the level of the variance or
variability of a measure. That is, across a set of scores, we assume that:
var(X) = var(T) + var(eX)
In more human terms this means that the variability of your measure is the sum of the variability
due to true score and the variability due to random error. This will have important implications
when we consider some of the more advanced models for adjusting for errors in measurement.
Why is true score theory important? For one thing, it is a simple yet powerful model for
measurement. It reminds us that most measurement has an error component. Second, true score
theory is the foundation of reliability theory. A measure that has no random error (i.e., is all true
score) is perfectly reliable; a measure that has no true score (i.e., is all random error) has zero
reliability. Third, true score theory can be used in computer simulations as the basis for
generating "observed" scores with certain known properties.
You should know that the true score model is not the only measurement model available.
measurement theorists continue to come up with more and more complex models that they think
represent reality even better. But these models are complicated enough that they lie outside the
boundaries of this document. In any event, true score theory should give you an idea of why
measurement models are important at all and how they can be used as the basis for defining key
research ideas.

Measurement Error
The true score theory is a good simple model for measurement, but it may not always be an
accurate reflection of reality. In particular, it assumes that any observation is composed of the
true value plus some random error value. But is that reasonable? What if all error is not random?
Isn't it possible that some errors are systematic, that they hold across most or all of the members
of a group? One way to deal with this notion is to revise the simple true score model by dividing
the error component into two subcomponents, random error and systematic error. here, we'll
look at the differences between these two types of errors and try to diagnose their effects on our
research.
What is Random Error?
Random error is caused by any factors that randomly affect measurement of the variable across
the sample. For instance, each person's mood can inflate or deflate their performance on any
occasion. In a particular testing, some children may be feeling in a good mood and others may be
depressed. If mood affects their performance on the measure, it may artificially inflate the
observed scores for some children and artificially deflate them for others. The important thing
about random error is that it does not have any consistent effects across the entire sample.
Instead, it pushes observed scores up or down randomly. This means that if we could see all of
the random errors in a distribution they would have to sum to 0 -- there would be as many
negative errors as positive ones. The important property of random error is that it adds variability
to the data but does not affect average performance for the group. Because of this, random error
is sometimes considered noise.

What is Systematic Error?

Systematic error is caused by any factors that systematically affect measurement of the variable
across the sample. For instance, if there is loud traffic going by just outside of a classroom where
students are taking a test, this noise is liable to affect all of the children's scores -- in this case,
systematically lowering them. Unlike random error, systematic errors tend to be consistently
either positive or negative -- because of this, systematic error is sometimes considered to be bias
in measurement.
Reducing Measurement Error
So, how can we reduce measurement errors, random or systematic? One thing you can do is to
pilot test your instruments, getting feedback from your respondents regarding how easy or hard
the measure was and information about how the testing environment affected their performance.
Second, if you are gathering measures using people to collect the data (as interviewers or
observers) you should make sure you train them thoroughly so that they aren't inadvertently
introducing error. Third, when you collect the data for your study you should double-check the
data thoroughly. All data entry for computer analysis should be "double-punched" and verified.
This means that you enter the data twice, the second time having your data entry machine check
that you are typing the exact same data you did the first time. Fourth, you can use statistical
procedures to adjust for measurement error. These range from rather simple formulas you can
apply directly to your data to very complex modeling procedures for modeling the error and its
effects. Finally, one of the best things you can do to deal with measurement errors, especially
systematic errors, is to use multiple measures of the same construct. Especially if the different
measures don't share the same systematic errors, you will be able to triangulate across the
multiple measures and get a more accurate sense of what's going on.

Theory of Reliability
What is reliability? We hear the term used a lot in research contexts, but what does it really
mean? If you think about how we use the word "reliable" in everyday language, you might get a
hint. For instance, we often speak about a machine as reliable: "I have a reliable car." Or, news
people talk about a "usually reliable source". In both cases, the word reliable usually means
"dependable" or "trustworthy." In research, the term "reliable" also means dependable in a
general sense, but that's not a precise enough definition. What does it mean to have a dependable
measure or observation in a research context? The reason "dependable" is not a good enough
description is that it can be confused too easily with the idea of a valid measure (see
Measurement Validity). Certainly, when we speak of a dependable measure, we mean one that is
both reliable and valid. So we have to be a little more precise when we try to define reliability.
In research, the term reliability means "repeatability" or "consistency". A measure is considered
reliable if it would give us the same result over and over again (assuming that what we are
measuring isn't changing!).
Let's explore in more detail what it means to say that
a measure is "repeatable" or "consistent". We'll
begin by defining a measure that we'll arbitrarily
label X. It might be a person's score on a math
achievement test or a measure of severity of illness.
It is the value (numerical or otherwise) that we
observe in our study. Now, to see how repeatable or
consistent an observation is, we can measure it
twice. We'll use subscripts to indicate the first and
second observation of the same measure. If we
assume that what we're measuring doesn't change
between the time of our first and second observation,
we can begin to understand how we get at reliability. While we observe a score for what we're
measuring, we usually think of that score as consisting of two parts, the 'true' score or actual
level for the person on that measure, and the 'error' in measuring it (see True Score Theory).
It's important to keep in mind that we observe the X score -- we never actually see the true (T)
or error (e) scores. For instance, a student may get a score of 85 on a math achievement test.
That's the score we observe, an X of 85. But the reality might be that the student is actually
better at math than that score indicates. Let's say the student's true math ability is 89 (i.e., T=89).
That means that the error for that student is -4. What does this mean? Well, while the student's
true math ability may be 89, he/she may have had a bad day, may not have had breakfast, may
have had an argument, or may have been distracted while taking the test. Factors like these can
contribute to errors in measurement that make the student's observed ability appear lower than
their true or actual ability.
OK, back to reliability. If our measure, X, is reliable, we should find that if we measure or
observe it twice on the same persons that the scores are pretty much the same. But why would
they be the same? If you look at the figure you should see that the only thing that the two
observations have in common is their true scores, T. How do you know that? Because the error
scores (e1 and e2) have different subscripts indicating that they are different values. But the true
score symbol T is the same for both observations. What does this mean? That the two observed
scores, X1 and X2 are related only to the degree that the observations share true score. You
should remember that the error score is assumed to be random. Sometimes errors will lead you to
perform better on a test than your true ability (e.g., you had a good day guessing!) while other
times it will lead you to score worse. But the true score -- your true ability on that measure --
would be the same on both observations (assuming, of course, that your true ability didn't change
between the two measurement occasions).
With this in mind, we can now define reliability more precisely. Reliability is a ratio or fraction.
In layperson terms we might define this ratio as:
 true level on the measure

the entire measure

You might think of reliability as the proportion of "truth" in your measure. Now, we don't speak
of the reliability of a measure for an individual -- reliability is a characteristic of a measure that's
taken across individuals. So, to get closer to a more formal definition, let's restate the definition
above in terms of a set of observations. The easiest way to do this is to speak of the variance of
the scores. Remember that the variance is a measure of the spread or distribution of a set of
scores. So, we can now state the definition as:
the variance of the true score

the variance of the measure

We might put this into slightly more technical terms by using the abbreviated name for the
variance and our variable names:
var(T)

var(X)
We're getting to the critical part now. If you look at the equation above, you should recognize
that we can easily determine or calculate the bottom part of the reliability ratio -- it's just the
variance of the set of scores we observed (You remember how to calculate the variance, don't
you? It's just the sum of the squared deviations of the scores from their mean, divided by the
number of scores). But how do we calculate the variance of the true scores. We can't see the true
scores (we only see X)! Only God knows the true score for a specific observation. And, if we
can't calculate the variance of the true scores, we can't compute our ratio, which means we can't
compute reliability! Everybody got that? The bottom line is...
we can't compute reliability because we can't calculate the variance of the true scores
Great. So where does that leave us? If we can't compute reliability, perhaps the best we can do is
to estimate it. Maybe we can get an estimate of the variability of the true scores. How do we do
that? Remember our two observations, X1 and X2? We assume (using true score theory) that
these two observations would be related to each other to the degree that they share true scores.
So, let's calculate the correlation between X1 and X2. Here's a simple formula for the correlation:
covariance(X1, X2)

sd(X1) * sd(X2)
where the 'sd' stands for the standard deviation (which is the square root of the variance). If we
look carefully at this equation, we can see that the covariance, which simply measures the
"shared" variance between measures must be an indicator of the variability of the true scores
because the true scores in X1 and X2 are the only thing the two observations share! So, the top
part is essentially an estimate of var(T) in this context. And, since the bottom part of the
equation multiplies the standard deviation of one observation with the standard deviation of the
same measure at another time, we would expect that these two values would be the same (it is
the same measure we're taking) and that this is essentially the same thing as squaring the
standard deviation for either observation. But, the square of the standard deviation is the same
thing as the variance of the measure. So, the bottom part of the equation becomes the variance of
the measure (or var(X)). If you read this paragraph carefully, you should see that the correlation
between two observations of the same measure is an estimate of reliability.
It's time to reach some conclusions. We know from this discussion that we cannot calculate
reliability because we cannot measure the true score component of an observation. But we also
know that we can estimate the true score component as the covariance between two observations
of the same measure. With that in mind, we can estimate the reliability as the correlation between
two observations of the same measure. It turns out that there are several ways we can estimate
this reliability correlation. These are discussed in Types of Reliability.
There's only one other issue I want to address here. How big is an estimate of reliability? To
figure this out, let's go back to the equation given earlier:
var(T)

var(X)
and remember that because X = T + e, we can substitute in the bottom of the ratio:
var(T)

var(T) + var(e)
With this slight change, we can easily determine the range of a reliability estimate. If a measure
is perfectly reliable, there is no error in measurement -- everything we observe is true score.
Therefore, for a perfectly reliable measure, the equation would reduce to:
var(T)

var(T)
and reliability = 1. Now, if we have a perfectly unreliable measure, there is no true score -- the
measure is entirely error. In this case, the equation would reduce to:
0

var(e)
and the reliability = 0. From this we know that reliability will always range between 0 and 1. The
value of a reliability estimate tells us the proportion of variability in the measure attributable to
the true score. A reliability of .5 means that about half of the variance of the observed score is
attributable to truth and half is attributable to error. A reliability of .8 means the variability is
about 80% true ability and 20% error. And so on.

Types of Reliability
You learned in the Theory of Reliability that it's not possible to calculate reliability exactly.
Instead, we have to estimate reliability, and this is always an imperfect endeavor. Here, I want to
introduce the major reliability estimators and talk about their strengths and weaknesses.
There are four general classes of reliability estimates, each of which estimates reliability in a
different way. They are:
• Inter-Rater or Inter-Observer Reliability
Used to assess the degree to which different raters/observers give consistent
estimates of the same phenomenon.
• Test-Retest Reliability
Used to assess the consistency of a measure from one time to another.
• Parallel-Forms Reliability
Used to assess the consistency of the results of two tests constructed in the
same way from the same content domain.
• Internal Consistency Reliability
Used to assess the consistency of results across items within a test.
Let's discuss each of these in turn.
Inter-Rater or Inter-Observer Reliability
Whenever you use humans as a part of your measurement
procedure, you have to worry about whether the results
you get are reliable or consistent. People are notorious for
their inconsistency. We are easily distractible. We get
tired of doing repetitive tasks. We daydream. We
misinterpret.
So how do we determine whether two observers are being
consistent in their observations? You probably should
establish inter-rater reliability outside of the context of the
measurement in your study. After all, if you use data from
your study to establish reliability, and you find that reliability is low, you're kind of stuck.
Probably it's best to do this as a side study or pilot study. And, if your study goes on for a long
time, you may want to reestablish inter-rater reliability from time to time to assure that your
raters aren't changing.
There are two major ways to actually estimate inter-rater reliability. If your measurement
consists of categories -- the raters are checking off which category each observation falls in --
you can calculate the percent of agreement between the raters. For instance, let's say you had 100
observations that were being rated by two raters. For each observation, the rater could check one
of three categories. Imagine that on 86 of the 100 observations the raters checked the same
category. In this case, the percent of agreement would be 86%. OK, it's a crude measure, but it
does give an idea of how much agreement exists, and it works no matter how many categories
are used for each observation.
The other major way to estimate inter-rater reliability is appropriate when the measure is a
continuous one. There, all you need to do is calculate the correlation between the ratings of the
two observers. For instance, they might be rating the overall level of activity in a classroom on a
1-to-7 scale. You could have them give their rating at regular time intervals (e.g., every 30
seconds). The correlation between these ratings would give you an estimate of the reliability or
consistency between the raters.
You might think of this type of reliability as "calibrating" the observers. There are other things
you could do to encourage reliability between observers, even if you don't estimate it. For
instance, I used to work in a psychiatric unit where every morning a nurse had to do a ten-item
rating of each patient on the unit. Of course, we couldn't count on the same nurse being present
every day, so we had to find a way to assure that any of the nurses would give comparable
ratings. The way we did it was to hold weekly "calibration" meetings where we would have all of
the nurses ratings for several patients and discuss why they chose the specific values they did. If
there were disagreements, the nurses would discuss them and attempt to come up with rules for
deciding when they would give a "3" or a "4" for a rating on a specific item. Although this was
not an estimate of reliability, it probably went a long way toward improving the reliability
between raters.
Test-Retest Reliability
We estimate test-retest reliability when we administer the same test to the same sample on two
different occasions. This approach assumes that there is no substantial change in the construct
being measured between the two occasions. The amount of time allowed between measures is
critical. We know that if we measure the same thing twice that the correlation between the two
observations will depend in part by how much time elapses between the two measurement
occasions. The shorter the time gap, the higher the correlation; the longer the time gap, the lower
the correlation. This is because the two observations are related over time -- the closer in time we
get the more similar the factors that contribute to error. Since this correlation is the test-retest
estimate of reliability, you can obtain considerably different estimates depending on the interval.

Parallel-Forms Reliability
In parallel forms reliability you first have to create two parallel forms. One way to accomplish
this is to create a large set of questions that address the same construct and then randomly divide
the questions into two sets. You administer both instruments to the same sample of people. The
correlation between the two parallel forms is the estimate of reliability. One major problem with
this approach is that you have to be able to generate lots of items that reflect the same construct.
This is often no easy feat. Furthermore, this approach makes the assumption that the randomly
divided halves are parallel or equivalent. Even by chance this will sometimes not be the case.
The parallel forms approach is very similar to the split-half reliability described below. The
major difference is that parallel forms are constructed so that the two forms can be used
independent of each other and considered equivalent measures. For instance, we might be
concerned about a testing threat to internal validity. If we use Form A for the pretest and Form B
for the posttest, we minimize that problem. it would even be better if we randomly assign
individuals to receive Form A or B on the pretest and then switch them on the posttest. With
split-half reliability we have an instrument that we wish to use as a single measurement
instrument and only develop randomly split halves for purposes of estimating reliability.

Internal Consistency Reliability

In internal consistency reliability estimation we use our single measurement instrument
administered to a group of people on one occasion to estimate reliability. In effect we judge the
reliability of the instrument by estimating how well the items that reflect the same construct yield
similar results. We are looking at how consistent the results are for different items for the same
construct within the measure. There are a wide variety of internal consistency measures that can
be used.
Average Inter-item Correlation
The average inter-item correlation uses all of the items on our instrument that are designed to
measure the same construct. We first compute the correlation between each pair of items, as
illustrated in the figure. For example, if we have six items we will have 15 different item pairings
(i.e., 15 correlations). The average interitem correlation is simply the average or mean of all
these correlations. In the example, we find an average inter-item correlation of .90 with the
individual correlations ranging from .84 to .95.
Average Itemtotal Correlation
This approach also uses the inter-item correlations. In addition, we compute a total score for the
six items and use that as a seventh variable in the analysis. The figure shows the six item-to-total
correlations at the bottom of the correlation matrix. They range from .82 to .88 in this sample
analysis, with the average of these at .85.

Split-Half Reliability
In split-half reliability we randomly divide all items that purport to measure the same construct
into two sets. We administer the entire instrument to a sample of people and calculate the total
score for each randomly divided half. the split-half reliability estimate, as shown in the figure, is
simply the correlation between these two total scores. In the example it is .87.
Cronbach's Alpha (α )
Imagine that we compute one split-half reliability and then randomly divide the items into
another set of split halves and recompute, and keep doing this until we have computed all
possible split half estimates of reliability. Cronbach's Alpha is mathematically equivalent to the
average of all possible split-half estimates, although that's not how we compute it. Notice that
when I say we compute all possible split-half estimates, I don't mean that each time we go an
measure a new sample! That would take forever. Instead, we calculate all split-half estimates
from the same sample. Because we measured all of our sample on each of the six items, all we
have to do is have the computer analysis do the random subsets of items and compute the
resulting correlations. The figure shows several of the split-half estimates for our six item
example and lists them as SH with a subscript. Just keep in mind that although Cronbach's Alpha
is equivalent to the average of all possible split half correlations we would never actually
calculate it that way. Some clever mathematician (Cronbach, I presume!) figured out a way to
get the mathematical equivalent a lot more quickly.

Comparison of Reliability Estimators

Each of the reliability estimators has certain advantages and disadvantages. Inter-rater reliability
is one of the best ways to estimate reliability when your measure is an observation. However, it
requires multiple raters or observers. As an alternative, you could look at the correlation of
ratings of the same single observer repeated on two different occasions. For example, let's say
you collected videotapes of child-mother interactions and had a rater code the videos for how
often the mother smiled at the child. To establish inter-rater reliability you could take a sample of
videos and have two raters code them independently. To estimate test-retest reliability you could
have a single rater code the same videos on two different occasions. You might use the inter-
rater approach especially if you were interested in using a team of raters and you wanted to
establish that they yielded consistent results. If you get a suitably high inter-rater reliability you
could then justify allowing them to work independently on coding different videos. You might
use the test-retest approach when you only have a single rater and don't want to train any others.
On the other hand, in some studies it is reasonable to do both to help establish the reliability of
the raters or observers.
The parallel forms estimator is typically only used in situations where you intend to use the two
forms as alternate measures of the same thing. Both the parallel forms and all of the internal
consistency estimators have one major constraint -- you have to have multiple items designed to
measure the same construct. This is relatively easy to achieve in certain contexts like
achievement testing (it's easy, for instance, to construct lots of similar addition problems for a
math test), but for more complex or subjective constructs this can be a real challenge. If you do
have lots of items, Cronbach's Alpha tends to be the most frequently used estimate of internal
consistency.
The test-retest estimator is especially feasible in most experimental and quasi-experimental
designs that use a no-treatment control group. In these designs you always have a control group
that is measured on two occasions (pretest and posttest). the main problem with this approach is
that you don't have any information about reliability until you collect the posttest and, if the
reliability estimate is low, you're pretty much sunk.
Each of the reliability estimators will give a different value for reliability. In general, the test-
retest and inter-rater reliability estimates will be lower in value than the parallel forms and
internal consistency ones because they involve measuring at different times or with different
raters. Since reliability estimates are often used in statistical analyses of quasi-experimental
designs (e.g., the analysis of the nonequivalent group design), the fact that different estimates can
differ considerably makes the analysis even more complex.

Reliability & Validity

We often think of reliability and validity as separate ideas but, in fact, they're related to each
other. Here, I want to show you two ways you can think about their relationship.
One of my favorite metaphors for the relationship between reliability is that of the target. Think
of the center of the target as the concept that you are trying to measure. Imagine that for each
person you are measuring, you are taking a shot at the target. If you measure the concept
perfectly for a person, you are hitting the center of the target. If you don't, you are missing the
center. The more you are off for that person, the further you are from the center.
The figure above shows four possible situations. In the first one, you are hitting the target
consistently, but you are missing the center of the target. That is, you are consistently and
systematically measuring the wrong value for all respondents. This measure is reliable, but no
valid (that is, it's consistent but wrong). The second, shows hits that are randomly spread across
the target. You seldom hit the center of the target but, on average, you are getting the right
answer for the group (but not very well for individuals). In this case, you get a valid group
estimate, but you are inconsistent. Here, you can clearly see that reliability is directly related to
the variability of your measure. The third scenario shows a case where your hits are spread
across the target and you are consistently missing the center. Your measure in this case is neither
reliable nor valid. Finally, we see the "Robin Hood" scenario -- you consistently hit the center of
the target. Your measure is both reliable and valid (I bet you never thought of Robin Hood in
those terms before).
Another way we can think about the relationship between reliability and validity is shown in the
figure below. Here, we set up a 2x2 table. The columns of the table indicate whether you are
trying to measure the same or different concepts. The rows show whether you are using the same
or different methods of measurement. Imagine that we have two concepts we would like to
measure, student verbal and math ability. Furthermore, imagine that we can measure each of
these in two ways. First, we can use a written, paper-and-pencil exam (very much like the SAT
or GRE exams). Second, we can ask the student's classroom teacher to give us a rating of the
student's ability based on their own classroom observation.

The first cell on the upper left shows the comparison of the verbal written test score with the
verbal written test score. But how can we compare the same measure with itself? We could do
this by estimating the reliability of the written test through a test-retest correlation, parallel
forms, or an internal consistency measure (See Types of Reliability). What we are estimating in
this cell is the reliability of the measure.
The cell on the lower left shows a comparison of the verbal written measure with the verbal
teacher observation rating. Because we are trying to measure the same concept, we are looking at
convergent validity (See Measurement Validity Types).
The cell on the upper right shows the comparison of the verbal written exam with the math
written exam. Here, we are comparing two different concepts (verbal versus math) and so we
would expect the relationship to be lower than a comparison of the same concept with itself (e.g.,
verbal versus verbal or math versus math). Thus, we are trying to discriminate between two
concepts and we would consider this discriminant validity.
Finally, we have the cell on the lower right. Here, we are comparing the verbal written exam
with the math teacher observation rating. Like the cell on the upper right, we are also trying to
compare two different concepts (verbal versus math) and so this is a discriminant validity
estimate. But here, we are also trying to compare two different methods of measurement (written
exam versus teacher observation rating). So, we'll call this very discriminant to indicate that we
would expect the relationship in this cell to be even lower than in the one above it.
The four cells incorporate the different values that we examine in the multitrait-multimethod
approach to estimating construct validity.
When we look at reliability and validity in this way, we see that, rather than being distinct, they
actually form a continuum. On one end is the situation where the concepts and methods of
measurement are the same (reliability) and on the other is the situation where concepts and
methods of measurement are different (very discriminant validity).

Levels of Measurement
The level of
measurement
refers to the
relationship
among the
values that are
assigned to the
attributes for a
variable. What
does that mean?
Begin with the
idea of the
variable, in this
example "party
affiliation."
That variable
has a number of attributes. Let's assume that in this particular election context the only relevant
attributes are "republican", "democrat", and "independent". For purposes of analyzing the results
of this variable, we arbitrarily assign the values 1, 2 and 3 to the three attributes. The level of
measurement describes the relationship among these three values. In this case, we simply are
using the numbers as shorter placeholders for the lengthier text terms. We don't assume that
higher values mean "more" of something and lower numbers signify "less". We don't assume the
the value of 2 means that democrats are twice something that republicans are. We don't assume
that republicans are in first place or have the highest priority just because they have the value of
1. In this case, we only use the values as a shorter name for the attribute. Here, we would
describe the level of measurement as "nominal".
Why is Level of Measurement Important?
First, knowing the level of measurement helps you decide how to interpret the data from that
variable. When you know that a measure is nominal (like the one just described), then you know
that the numerical values are just short codes for the longer names. Second, knowing the level of
measurement helps you decide what statistical analysis is appropriate on the values that were
assigned. If a measure is nominal, then you know that you would never average the data values
or do a t-test on the data.
There are typically four levels of measurement that are defined:
• Nominal
• Ordinal
• Interval
• Ratio
In nominal measurement the numerical values just "name" the attribute uniquely. No ordering of
the cases is implied. For example, jersey numbers in basketball are measures at the nominal
level. A player with number 30 is not more of anything than a player with number 15, and is
certainly not twice whatever number 15 is.
In ordinal measurement the attributes can be rank-ordered. Here, distances between attributes do
not have any meaning. For example, on a survey you might code Educational Attainment as
0=less than H.S.; 1=some H.S.; 2=H.S. degree; 3=some college; 4=college degree; 5=post
college. In this measure, higher numbers mean more education. But is distance from 0 to 1 same
as 3 to 4? Of course not. The
interval between values is not
interpretable in an ordinal
measure.
In interval measurement the
distance between attributes
does have meaning. For
example, when we measure
temperature (in Fahrenheit), the
distance from 30-40 is same as
distance from 70-80. The
interval between values is
interpretable. Because of this, it
makes sense to compute an
average of an interval variable,
where it doesn't make sense to
do so for ordinal scales. But
note that in interval measurement ratios don't make any sense - 80 degrees is not twice as hot as
40 degrees (although the attribute value is twice as large).
Finally, in ratio measurement there is always an absolute zero that is meaningful. This means
that you can construct a meaningful fraction (or ratio) with a ratio variable. Weight is a ratio
variable. In applied social research most "count" variables are ratio, for example, the number of
clients in past six months. Why? Because you can have zero clients and because it is meaningful
to say that "...we had twice as many clients in the past six months as we did in the previous six
months."
It's important to recognize that there is a hierarchy implied in the level of measurement idea. At
lower levels of measurement, assumptions tend to be less restrictive and data analyses tend to be
less sensitive. At each level up the hierarchy, the current level includes all of the qualities of the
one below it and adds something new. In general, it is desirable to have a higher level of
measurement (e.g., interval or ratio) rather than a lower one (nominal or ordinal).

Survey Research
Survey research is one of the most important areas of measurement in applied social research.
The broad area of survey research encompasses any measurement procedures that involve asking
questions of respondents. A "survey" can be anything form a short paper-and-pencil feedback
form to an intensive one-on-one in-depth interview.
We'll begin by looking at the different types of surveys that are possible. These are roughly
divided into two broad areas: Questionnaires and Interviews. Next, we'll look at how you select
the survey method that is best for your situation. Once you've selected the survey method, you
have to construct the survey itself. Here, we will be address a number of issues including: the
different types of questions; decisions about question content; decisions about question wording;
decisions about response format; and, question placement and sequence in your instrument. We
turn next to some of the special issues involved in administering a personal interview. Finally,
we'll consider some of the advantages and disadvantages of survey methods
Types of Surveys
Surveys can be divided into two broad categories: the questionnaire and the interview.
Questionnaires are usually paper-and-pencil instruments that the respondent completes.
Interviews are completed by the interviewer based on the respondent says. Sometimes, it's hard
to tell the difference between a questionnaire and an interview. For instance, some people think
that questionnaires always ask short closed-ended questions while interviews always ask broad
open-ended ones. But you will see questionnaires with open-ended questions (although they do
tend to be shorter than in interviews) and there will often be a series of closed-ended questions
asked in an interview.
Survey research has changed dramatically in the last ten years. We have automated telephone
surveys that use random dialing methods. There are computerized kiosks in public places that
allows people to ask for input. A whole new variation of group interview has evolved as focus
group methodology. Increasingly, survey research is tightly integrated with the delivery of
service. Your hotel room has a survey on the desk. Your waiter presents a short customer
satisfaction survey with your check. You get a call for an interview several days after your last
call to a computer company for technical assistance. You're asked to complete a short survey
when you visit a web site. Here, I'll describe the major types of questionnaires and interviews,
keeping in mind that technology is leading to rapid evolution of methods. We'll discuss the
relative advantages and disadvantages of these different survey types in Advantages and
Disadvantages of Survey Methods.
Questionnaires
When most people think of questionnaires, they think of the mail
survey. All of us have, at one time or another, received a questionnaire
in the mail. There are many advantages to mail surveys. They are
relatively inexpensive to administer. You can send the exact same
instrument to a wide number of people. They allow the respondent to fill it out at their own
convenience. But there are some disadvantages as well. Response rates from mail surveys are
often very low. And, mail questionnaires are not the best vehicles for asking for detailed written
responses.
A second type is the group administered questionnaire. A sample of
respondents is brought together and asked to respond to a structured
sequence of questions. Traditionally, questionnaires were administered
in group settings for convenience. The researcher could give the
questionnaire to those who were present and be fairly sure that there
would be a high response rate. If the respondents were unclear about
the meaning of a question they could ask for clarification. And, there
were often organizational settings where it was relatively easy to assemble the group (in a
company or business, for instance).
What's the difference between a group administered questionnaire and a group interview or focus
group? In the group administered questionnaire, each respondent is handed an instrument and
asked to complete it while in the room. Each respondent completes an instrument. In the group
interview or focus group, the interviewer facilitates the session. People work as a group, listening
to each other's comments and answering the questions. Someone takes notes for the entire group
-- people don't complete an interview individually.
 A less familiar type of questionnaire is the household drop-off survey.
In this approach, a researcher goes to the respondent's home or business
and hands the respondent the instrument. In some cases, the respondent
is asked to mail it back or the interview returns to pick it up. This
approach attempts to blend the advantages of the mail survey and the
group administered questionnaire. Like the mail survey, the respondent
can work on the instrument in private, when it's convenient. Like the
group administered questionnaire, the interviewer makes personal contact with the respondent --
they don't just send an impersonal survey instrument. And, the respondent can ask questions
about the study and get clarification on what is to be done. Generally, this would be expected to
increase the percent of people who are willing to respond.
Interviews
Interviews are a far more personal form of research than questionnaires. In
the personal interview, the interviewer works directly with the respondent.
Unlike with mail surveys, the interviewer has the opportunity to probe or
ask follow-up questions. And, interviews are generally easier for the
respondent, especially if what is sought is opinions or impressions.
Interviews can be very time consuming and they are resource intensive. The
interviewer is considered a part of the measurement instrument and
interviewers have to be well trained in how to respond to any contingency.
Almost everyone is familiar with the telephone interview. Telephone
interviews enable a researcher to gather information rapidly. Most of
the major public opinion polls that are reported were based on
telephone interviews. Like personal interviews, they allow for some
personal contact between the interviewer and the respondent. And,
they allow the interviewer to ask follow-up questions. But they also
have some major disadvantages. Many people don't have publicly-
listed telephone numbers. Some don't have telephones. People often don't like the intrusion of a
call to their homes. And, telephone interviews have to be relatively short or people will feel
imposed upon.

Selecting the Survey Method

Selecting the type of survey you are going to use is one of the most critical decisions in many
social research contexts. You'll see that there are very few simple rules that will make the
decision for you -- you have to use your judgment to balance the advantages and disadvantages
of different survey types. Here, all I want to do is give you a number of questions you might ask
that can help guide your decision.
Population Issues
The first set of considerations have to do with the population and its accessibility.
• Can the population be enumerated?
For some populations, you have a complete listing of the units that will be sampled. For others,
such a list is difficult or impossible to compile. For instance, there are complete listings of
registered voters or person with active drivers licenses. But no one keeps a complete list of
homeless people. If you are doing a study that requires input from homeless persons, you are
very likely going to need to go and find the respondents personally. In such contexts, you can
pretty much rule out the idea of mail surveys or telephone interviews.
• Is the population literate?
Questionnaires require that your respondents can read. While this might seem initially like a
reasonable assumption for many adult populations, we know from recent research that the
instance of adult illiteracy is alarmingly high. And, even if your respondents can read to some
degree, your questionnaire may contain difficult or technical vocabulary. Clearly, there are some
populations that you would expect to be illiterate. Young children would not be good targets for
questionnaires.
• Are there language issues?
We live in a multilingual world. Virtually every society has members who speak other than the
predominant language. Some countries (like Canada) are officially multilingual. And, our
increasingly global economy requires us to do research that spans countries and language groups.
Can you produce multiple versions of your questionnaire? For mail instruments, can you know in
advance the language your respondent speaks, or do you send multiple translations of your
instrument? Can you be confident that important connotations in your instrument are not
culturally specific? Could some of the important nuances get lost in the process of translating
your questions?
• Will the population cooperate?
People who do research on immigration issues have a difficult methodological problem. They
often need to speak with undocumented immigrants or people who may be able to identify others
who are. Why would we expect those respondents to cooperate? Although the researcher may
mean no harm, the respondents are at considerable risk legally if information they divulge should
get into the hand of the authorities. The same can be said for any target group that is engaging in
illegal or unpopular activities.
• What are the geographic restrictions?
Is your population of interest dispersed over too broad a geographic range for you to study
feasibly with a personal interview? It may be possible for you to send a mail instrument to a
nationwide sample. You may be able to conduct phone interviews with them. But it will almost
certainly be less feasible to do research that requires interviewers to visit directly with
respondents if they are widely dispersed.
Sampling Issues

The sample is the actual group you will have to contact in some way. There are several important
sampling issues you need to consider when doing survey research.
• What data is available?
What information do you have about your sample? Do you know their current addresses? Their
current phone numbers? Are your contact lists up to date?
• Can respondents be found?
Can your respondents be located? Some people are very busy. Some travel a lot. Some work the
night shift. Even if you have an accurate phone or address, you may not be able to locate or make
contact with your sample.
• Who is the respondent?
Who is the respondent in your study? Let's say you draw a sample of households in a small city.
A household is not a respondent. Do you want to interview a specific individual? Do you want to
talk only to the "head of household" (and how is that person defined)? Are you willing to talk to
any member of the household? Do you state that you will speak to the first adult member of the
household who opens the door? What if that person is unwilling to be interviewed but someone
else in the house is willing? How do you deal with multi-family households? Similar problems
arise when you sample groups, agencies, or companies. Can you survey any member of the
organization? Or, do you only want to speak to the Director of Human Resources? What if the
person you would like to interview is unwilling or unable to participate? Do you use another
member of the organization?
• Can all members of population be sampled?
If you have an incomplete list of the population (i.e., sampling frame) you may not be able to
sample every member of the population. Lists of various groups are extremely hard to keep up to
date. People move or change their names. Even though they are on your sampling frame listing,
you may not be able to get to them. And, it's possible they are not even on the list.
• Are response rates likely to be a problem?
Even if you are able to solve all of the other population and sampling problems, you still have to
deal with the issue of response rates. Some members of your sample will simply refuse to
respond. Others have the best of intentions, but can't seem to find the time to send in your
questionnaire by the due date. Still others misplace the instrument or forget about the
appointment for an interview. Low response rates are among the most difficult of problems in
survey research. They can ruin an otherwise well-designed survey effort.
Question Issues

Sometimes the nature of what you want to ask respondents will determine the type of survey you
select.
• What types of questions can be asked?
Are you going to be asking personal questions? Are you going to need to get lots of detail in the
responses? Can you anticipate the most frequent or important types of responses and develop
reasonable closed-ended questions?
• How complex will the questions be?
Sometimes you are dealing with a complex subject or topic. The questions you want to ask are
going to have multiple parts. You may need to branch to sub-questions.
• Will screening questions be needed?
A screening question may be needed to determine whether the respondent is qualified to answer
your question of interest. For instance, you wouldn't want to ask someone their opinions about a
specific computer program without first "screening" them to find out whether they have any
experience using the program. Sometimes you have to screen on several variables (e.g., age,
gender, experience). The more complicated the screening, the less likely it is that you can rely on
paper-and-pencil instruments without confusing the respondent.
• Can question sequence be controlled?
Is your survey one where you can construct in advance a reasonable sequence of questions? Or,
are you doing an initial exploratory study where you may need to ask lots of follow-up questions
that you can't easily anticipate?
• Will lengthy questions be asked?
If your subject matter is complicated, you may need to give the respondent some detailed
background for a question. Can you reasonably expect your respondent to sit still long enough in
a phone interview to ask your question?
• Will long response scales be used?
If you are asking people about the different computer equipment they use, you may have to have
a lengthy response list (CD-ROM drive, floppy drive, mouse, touch pad, modem, network
connection, external speakers, etc.). Clearly, it may be difficult to ask about each of these in a
short phone interview.
Content Issues
The content of your study can also pose challenges for the different survey types you might
utilize.
• Can the respondents be expected to know about the issue?
If the respondent does not keep up with the news (e.g., by reading the newspaper, watching
television news, or talking with others), they may not even know about the news issue you want
to ask them about. Or, if you want to do a study of family finances and you are talking to the
spouse who doesn't pay the bills on a regular basis, they may not have the information to answer
your questions.
• Will respondent need to consult records?
Even if the respondent understands what you're asking about, you may need to allow them to
consult their records in order to get an accurate answer. For instance, if you ask them how much
money they spent on food in the past month, they may need to look up their personal check and
credit card records. In this case, you don't want to be involved in an interview where they would
have to go look things up while they keep you waiting (they wouldn't be comfortable with that).
Bias Issues
People come to the research endeavor with their own sets of biases and prejudices. Sometimes,
these biases will be less of a problem with certain types of survey approaches.
• Can social desirability be avoided?
Respondents generally want to "look good" in the eyes of others. None of us likes to look like we
don't know an answer. We don't want to say anything that would be embarrassing. If you ask
people about information that may put them in this kind of position, they may not tell you the
truth, or they may "spin" the response so that it makes them look better. This may be more of a
problem in an interview situation where they are face-to face or on the phone with a live
interviewer.
• Can interviewer distortion and subversion be controlled?
Interviewers may distort an interview as well. They may not ask questions that make them
uncomfortable. They may not listen carefully to respondents on topics for which they have
strong opinions. They may make the judgment that they already know what the respondent
would say to a question based on their prior responses, even though that may not be true.
• Can false respondents be avoided?
With mail surveys it may be difficult to know who actually responded. Did the head of
household complete the survey or someone else? Did the CEO actually give the responses or
instead pass the task off to a subordinate? Is the person you're speaking with on the phone
actually who they say they are? At least with personal interviews, you have a reasonable chance
of knowing who you are speaking with. In mail surveys or phone interviews, this may not be the
case.
Administrative Issues
Last, but certainly not least, you have to consider the feasibility of the survey method for your
study.
• costs
Cost is often the major determining factor in selecting survey type. You might prefer to do
personal interviews, but can't justify the high cost of training and paying for the interviewers.
You may prefer to send out an extensive mailing but can't afford the postage to do so.
• facilities
Do you have the facilities (or access to them) to process and manage your study? In phone
interviews, do you have well-equipped phone surveying facilities? For focus groups, do you have
a comfortable and accessible room to host the group? Do you have the equipment needed to
record and transcribe responses?
• time
Some types of surveys take longer than others. Do you need responses immediately (as in an
overnight public opinion poll)? Have you budgeted enough time for your study to send out mail
surveys and follow-up reminders, and to get the responses back by mail? Have you allowed for
enough time to get enough personal interviews to justify that approach?
• personnel
Different types of surveys make different demands of personnel. Interviews require interviewers
who are motivated and well-trained. Group administered surveys require people who are trained
in group facilitation. Some studies may be in a technical area that requires some degree of
expertise in the interviewer.
Clearly, there are lots of issues to consider when you are selecting which type of survey you wish
to use in your study. And there is no clear and easy way to make this decision in many contexts.
There may not be one approach which is clearly the best. You may have to make tradeoffs of
advantages and disadvantages. There is judgment involved. Two expert researchers may, for the
very same problem or issue, select entirely different survey methods. But, if you select a method
that isn't appropriate or doesn't fit the context, you can doom a study before you even begin
designing the instruments or questions themselves.

Constructing the Survey

Constructing a survey instrument is an art in itself. There are numerous small decisions that must
be made -- about content, wording, format, placement -- that can have important consequences
for your entire study. While there's no one perfect way to accomplish this job, we do have lots of
advice to offer that might increase your chances of developing a better final product.
First of all you'll learn about the two major types of surveys that exist, the questionnaire and the
interview and the different varieties of each. Then you'll see how to write questions for surveys.
There are three areas involved in writing a question:
• determining the question content, scope and purpose
• choosing the response format that you use for collecting information from the respondent
• figuring out how to word the question to get at the issue of interest
Finally, once you have your questions written, there is the issue of how best to place them in
your survey.
You'll see that although there are many aspects of survey construction that are just common
sense, if you are not careful you can make critical errors that have dramatic effects on your
results.

Types Of Questions
Survey questions can be divided into two broad types: structured and unstructured. From an
instrument design point of view, the structured questions pose the greater difficulties (see
Decisions About the Response Format). From a content perspective, it may actually be more
difficult to write good unstructured questions. Here, I'll discuss the variety of structured
questions you can consider for your survey (we'll discuss unstructured questioning more under
Interviews).
Dichotomous Questions
When a question has two possible responses, we consider it dichotomous. Surveys often use
dichotomous questions that ask for a Yes/No, True/False or Agree/Disagree response. There are
a variety of ways to lay these questions out on a questionnaire:

Questions Based on Level Of Measurement

We can also classify questions in terms of their level of measurement. For instance, we might
measure occupation using a nominal question. Here, the number next to each response has no
meaning except as a placeholder for that response. The choice of a "2" for a lawyer and a "1" for
a truck driver is arbitrary -- from the numbering system used we can't infer that a lawyer is
"twice" something that a truck driver is.
We might ask respondents to rank order their preferences for presidential candidates using an
ordinal question:

We want the respondent to put a 1, 2, 3 or 4 next to the candidate, where 1 is the respondent's
first choice. Note that this could get confusing. We might want to state the prompt more
explicitly so the respondent knows we want a number from one to 4 (the respondent might check
their favorite candidate, or assign higher numbers to candidates they prefer more instead of
understanding that we want rank ordering).
We can also construct survey questions that attempt to measure on an interval level. One of the
most common of these types is the traditional 1-to-5 rating (or 1-to-7, or 1-to-9, etc.). This is
sometimes referred to as a Likert response scale (see Likert Scaling). Here, we see how we
might ask an opinion question on a 1-to-5 bipolar scale (it's called bipolar because there is a
neutral point and the two ends of the scale are at opposite positions of the opinion):

Another interval question uses an approach called the semantic differential. Here, an object is
assessed by the respondent on a set of bipolar adjective pairs (using 5-point rating scale):
Finally, we can also get at interval measures by using what is called a cumulative or Guttman
scale (see Guttman Scaling). Here, the respondent checks each item with which they agree. The
items themselves are constructed so that they are cumulative -- if you agree to one, you probably
agree to all of the ones above it in the list:

Filter or Contingency Questions

Sometimes you have to ask the respondent one question in order to determine if they are
qualified or experienced enough to answer a subsequent one. This requires using a filter or
contingency question. For instance, you may want to ask one question if the respondent has
ever smoked marijuana and a different question if they have not. in this case, you would have to
construct a filter question to determine whether they've ever smoked marijuana:
Filter questions can get very complex. Sometimes, you have to have multiple filter questions in
order to direct your respondents to the correct subsequent questions. There are a few conventions
you should keep in mind when using filters:
• try to avoid having more than three levels (two jumps) for any question
Too many jumps will confuse the respondent and may discourage them from continuing with the
survey.
• if only two levels, use graphic to jump (e.g., arrow and box)
The example above shows how you can make effective use of an arrow and box to help direct the
respondent to the correct subsequent question.
• if possible, jump to a new page
If you can't fit the response to a filter on a single page, it's probably best to be able to say
something like "If YES, please turn to page 4" rather that "If YES, please go to Question 38"
because the respondent will generally have an easier time finding a page than a specific question.

Question Content
For each question in your survey, you should ask yourself how well it addresses the content you
are trying to get at. Here are some content-related questions you can ask about your survey
questions.
Is the Question Necessary/Useful?
Examine each question to see if you need to ask it at all and if you need to ask it at the level of
detail you currently have.
 • Do you need the age of each child or just the number of children under 16?
• Do you need to ask income or can you estimate?
Are Several Questions Needed?
This is the classic problem of the double-barreled question. You should think about splitting
each of the following questions into two separate ones. You can often spot these kinds of
problems by looking for the conjunction "and" in your question.
• What are your feelings towards African-Americans and Hispanic-Americans?
• What do you think of proposed changes in benefits and hours?
Another reason you might need more than one question is that the question you ask does not
cover all possibilities. For instance, if you ask about earnings, the respondent might not mention
all income (e.g., dividends, gifts). Or, if you ask the respondents if they're in favor of public TV,
they might not understand that you're asking generally. They may not be in favor of public TV
for themselves (they never watch it), but might favor it very much for their children (who watch
Sesame Street regularly). You might be better off asking two questions, one for their own
viewing and one for other members of their household.
Sometimes you need to ask additional questions because your question does not give you
enough context to interpret the answer. For instance, if you ask about attitudes towards
Catholics, can you interpret this without finding out about their attitudes towards religion in
general, or other religious groups?
At times, you need to ask additional questions because your question does not determine the
intensity of the respondent's attitude or belief. For example, if they say they support public TV,
you probably should also ask them whether they ever watch it or if they would be willing to have
their tax dollars spent on it. It's one thing for a respondent to tell you they support something.
But the intensity of that response is greater if they are willing to back their sentiment of support
with their behavior.
Do Respondents Have the Needed Information?
Look at each question in your survey to see whether the respondent is likely to have the
necessary information to be able to answer the question. For example, let's say you want to ask
the question:
Do you think Dean Rusk acted correctly in the Bay of Pigs crisis?
The respondent won't be able to answer this question if they have no idea who Dean Rusk was or
what the Bay of Pigs crisis was. In surveys of television viewing, you cannot expect that the
respondent can answer questions about shows they have never watched. You should ask a filter
question first (e.g., Have you ever watched the show ER?) before asking them their opinions
about it.
Does the Question Need to be More Specific?
Sometimes we ask our questions too generally and the information we obtain is more difficult to
interpret. For example, let's say you want to find out respondent's opinions about a specific book.
You could ask them
How well did you like the book?
on some scale ranging from "Not At All" to "Extremely Well." But what would their response
mean? What does it mean to say you liked a book very well? Instead, you might as questions
designed to be more specific like:
Did you recommend the book to others?
or
Did you look for other books by that author?
Is Question Sufficiently General?
You can err in the other direction as well by being too specific. For instance, if you ask someone
to list the televisions program they liked best in the past week, you could get a very different
answer than if you asked them which show they've enjoyed most over the past year. Perhaps a
show they don't usually like had a great episode in the past week, or their show was preempted
by another program.
Is Question Biased or Loaded?
One danger in question-writing is that your own biases and blind-spots may affect the wording
(see Decisions About Question Wording). For instance, you might generally be in favor of tax
cuts. If you ask a question like:
What do you see as the benefits of a tax cut?
you're only asking about one side of the issue. You might get a very different picture of the
respondents' positions if you also asked about the disadvantages of tax cuts. The same thing
could occur if you are in favor of public welfare and you ask:
What do you see as the disadvantages of eliminating welfare?
without also asking about the potential benefits.
Will Respondent Answer Truthfully?
For each question on your survey, ask yourself whether the respondent will have any difficulty
answering the question truthfully. If there is some reason why they may not, consider rewording
the question. For instance, some people are sensitive about answering questions about their exact
age or income. In this case, you might give them response brackets to choose from (e.g.,
between 30 and 40 years old, between $50,000 and $100,000 annual income). Sometimes even
bracketed responses won't be enough. Some people do not like to share how much money they
give to charitable causes (they may be afraid of being solicited even more). No matter how you
word the question, they would not be likely to tell you their contribution rate. But sometimes you
can do this by posing the question in terms of a hypothetical projective respondent (a little bit
like a projective test). In this case, you might get reasonable estimates if you ask the respondent
how much money "people you know" typically give in a year to charitable causes. Finally, you
can sometimes dispense with asking a question at all if you can obtain the answer unobtrusively
(see Unobtrusive Measures). If you are interested in finding out what magazines the respondent
reads, you might instead tell them you are collecting magazines for a recycling drive and ask if
they have any old ones to donate (of course, you have to consider the ethical implications of such
deception!).

Response Format
The response format is how you collect the answer from the respondent. Let's start with a simple
distinction between what we'll call unstructured response formats and structured response
formats. [On this page, I'll use standard web-based form fields to show you how various
response formats might look on the web. If you want to see how these are generated, select the
View Source option on your web browser.]
Top of Form

Structured Response Formats

Structured formats help the respondent to respond more easily and help the researcher to
accumulate and summarize responses more efficiently. But, they can also constrain the
respondent and limit the researcher's ability to understand what the respondent really means.
There are many different structured response formats, each with its own strengths and
weaknesses. We'll review the major ones here.
Fill-In-The-Blank. One of the simplest response formats is a blank line. A blank line can be
used for a number of different response types. For instance:
Please enter your gender:
_____ Male
_____ Female
Here, the respondent would probably put a check mark or an X next to the response. This is also
an example of a dichotomous response, because it only has two possible values. Other common
dichotomous responses are True/False and Yes/No. Here's another common use of a fill-in-the-
blank response format:
Please enter your preference for the following candidates where '1' = your first choice, '2' =
your second choice, and so on.
_____ Robert Dole
_____ Colin Powell
_____ Bill Clinton
_____ Al Gore
In this example, the respondent writes a number in each blank. Notice that here, we expect the
respondent to place a number on every blank, whereas in the previous example, we expect to
respondent to choose only one. Then, of course, there's the classic:
NAME: ________________________
And here's the same fill-in-the-blank response item in web format:

NAME:
Of course, there's always the classic fill-in-the-blank test item:

One of President Lincoln's most famous speeches, the Address, only lasted a few
minutes when delivered.
Check The Answer. The respondent places a check next to the response(s). The simplest form
would be the example given above where we ask the person to indicate their gender. Sometimes,
we supply a box that the person can fill in with an 'X' (which is sort of a variation on the check
mark. Here's a web version of the checkbox:
Please check if you have the following item on the computer you use most:

modem

printer

CD-ROM drive

joystick

scanner
Notice that in this example, it is possible for you to check more than one response. By
convention, we usually use the checkmark format when we want to allow the respondent to
select multiple items.
We sometimes refer to this as a multi-option variable. You have to be careful when you analyze
data from a multi-option variable. Because the respondent can select any of the options, you have
to treat this type of variable in your analysis as though each option is a separate variable. For
instance, for each option we would normally enter either a '0' if the respondent did not check it or
a '1' if the respondent did check it. For the example above, if the respondent had only a modem
and CD-ROM drive, we would enter the sequence 1, 0, 1, 0, 0. There is a very important reason
why you should code this variable as either 0 or 1 when you enter the data. If you do, and you
want to determine what percent of your sample has a modem, all you have to do is compute the
average of the 0's and 1's for the modem variable. For instance, if you have 10 respondents and
only 3 have a modem, the average would be 3/10 = .30 or 30%, which is the percent who
checked that item.
The example above is also a good example of a checklist item. Whenever you use a checklist,
you want to be sure that you ask the following questions:
• Are all of the alternatives covered?
• Is the list of reasonable length?
• Is the wording impartial?
• Is the form of the response easy, uniform?
Sometimes you may not be sure that you have covered all of the possible responses in a
checklist. If that is the case, you should probably allow the respondent to write in any other
options that may apply.
Circle The Answer. Sometimes the respondent is asked to circle an item to indicate their
response. Usually we are asking them to circle a number. For instance, we might have the
following:
In computer contexts, it's not feasible to have respondents circle a response. In this case, we tend
to use an option button:
Capital punishment is the best way to deal with convicted murderers.

Strongly Disagree Neutral Agree Strongly

Disagree Agree

Notice that you can only check one option at a time. The rule of thumb is that you ask someone
to circle an item or click on a button when you only want them to be able to select one of the
options. In contrast to the multi-option variable described above, we refer to this type of item as
a single-option variable -- even though the respondent has multiple choices, they can only select
one of them. We would analyze this as a single variable that can take the integer values from 1 to
5.
Unstructured Response Formats
While there is a wide variety of structured response formats, there are relatively few unstructured
ones. What is an unstructured response format? Generally, it's written text. If the respondent (or
interviewer) writes down text as the response, you've got an unstructured response format. These
can vary from short comment boxes to the transcript of an interview.
In almost every short questionnaire, there's one or more short text field questions. One of the
most frequent goes something like this:
Actually, there's really not much more to text-based response formats of this type than writing
the prompt and allowing enough space for a reasonable response.
Transcripts are an entirely different matter. There, the transcriber has to decide whether to
transcribe every word or only record major ideas, thoughts, quotes, etc. In detailed transcriptions,
you may also need to distinguish different speakers (e.g., the interviewer and respondent) and
have a standard convention for indicating comments about what's going on in the interview,
including non-conversational events that take place and thoughts of the interviewer.

Question Wording
One of the major difficulty in writing good survey questions is getting the wording right. Even
slight wording differences can confuse the respondent or lead to incorrect interpretations of the
question. Here, I outline some questions you can ask about how you worded each of your survey
questions.
Can the Question be Misunderstood?
The survey author has to always be on the lookout for questions that could be misunderstood or
confusing. For instance, if you ask a person for their nationality, it might not be clear what you
want (Do you want someone from Malaysia to say Malaysian, Asian, or Pacific Islander?). Or, if
you ask for marital status, do you want someone to say simply that they are either married or no
married? Or, do you want more detail (like divorced, widow/widower, etc.)?
Some terms are just to vague to be useful. For instance, if you ask a question about the "mass
media," what do you mean? The newspapers? Radio? Television?
Here's one of my favorites. Let's say you want to know the following:
What kind of headache remedy do you use?
Do you want to know what brand name medicine they take? Do you want to know about "home"
remedies? Are you asking whether they prefer a pill, capsule or caplet?
What Assumptions Does the Question Make?
Sometimes we don't stop to consider how a question will appear from the respondent's point-of-
view. We don't think about the assumptions behind our questions. For instance, if you ask what
social class someone's in, you assume that they know what social class is and that they think of
themselves as being in one. In this kind of case, you may need to use a filter question first to
determine whether either of these assumptions is true.
Is the time frame specified?
Whenever you use the words "will", "could", "might", or "may" in a question, you might suspect
that the question asks a time-related question. Be sure that, if it does, you have specified the time
frame precisely. For instance, you might ask:
Do you think Congress will cut taxes?
or something like
Do you think Congress could successfully resist tax cuts?
Neither of these questions specifies a time frame.
How personal is the wording?
With a change of just a few words, a question can go from being relatively impersonal to probing
into your private perspectives. Consider the following three questions, each of which asks about
the respondent's satisfaction with working conditions:
• Are working conditions satisfactory or not satisfactory in the plant where you work?
• Do you feel that working conditions satisfactory or not satisfactory in the plant where you
work?
• Are you personally satisfied with working conditions in the plant where you work?
The first question is stated from a fairly detached, objective viewpoint. The second asks how you
"feel." The last asks whether you are "personally satisfied." Be sure the questions in your survey
are at an appropriate level for your context. And, be sure there is consistency in this across
questions in your survey.
Is the wording too direct?
There are times when asking a question too directly may be too threatening or disturbing for
respondents. For instance, consider a study where you want to discuss battlefield experiences
with former soldiers who experienced trauma. Examine the following three question options:
• How did you feel about being in the war?
• How well did the equipment hold up in the field?
• How well were new recruits trained?
The first question may be too direct. For this population it may elicit powerful negative emotions
based on their recollections. The second question is a less direct one. It asks about equipment in
the field, but, for this population, may also lead the discussion toward more difficult issues to
discuss directly. The last question is probably the least direct and least threatening. Bashing the
new recruits is standard protocol in almost any social context. The question is likely to get the
respondent talking, recounting anecdotes, without eliciting much stress. Of course, all of this
may simply be begging the question. If you are doing a study where the respondents may
experience high levels of stress because of the questions you ask, you should reconsider the
ethics of doing the study.
Other Wording Issues
The nuances of language guarantee that the task of the question writer will be endlessly complex.
Without trying to generate an exhaustive list, here are a few other questions to keep in mind:
• Does the question contain difficult or unclear terminology?
• Does the question make each alternative explicit?
• Is the wording objectionable?
• Is the wording loaded or slanted?

Question Placement
Decisions About Placement
One of the most difficult tasks facing the survey designer involves the ordering of questions.
Which topics should be introduced early in the survey, and which later? If you leave your most
important questions until the end, you may find that your respondents are too tired to give them
the kind of attention you would like. If you introduce them too early, they may not yet be ready
to address the topic, especially if it is a difficult or disturbing one. There are no easy answers to
these problems - you have to use your judgment. Whenever you think about question placement,
consider the following questions:
• Is the answer influenced by prior questions?
• Does question come too early or too late to arouse interest?
• Does the question receive sufficient attention?
The Opening Questions
Just as in other aspects of life, first impressions are important in survey work. The first few
questions you ask will determine the tone for the survey, and can help put your respondent at
ease. With that in mind, the opening few questions should, in general, be easy to answer. You
might start with some simple descriptive questions that will get the respondent rolling. You
should never begin your survey with sensitive or threatening questions.
Sensitive Questions
In much of our social research, we have to ask respondents about difficult or uncomfortable
subjects. Before asking such questions, you should attempt to develop some trust or rapport with
the respondent. Often, preceding the sensitive questions with some easier warm-up ones will
help. But, you have to make sure that the sensitive material does not come up abruptly or appear
unconnected with the rest of the survey. It is often helpful to have a transition sentence between
sections of your instrument to give the respondent some idea of the kinds of questions that are
coming. For instance, you might lead into a section on personal material with the transition:
In this next section of the survey, we'd like to ask you about your personal relationships.
Remember, we do not want you to answer any questions if you are uncomfortable doing so.
A Checklist of Considerations
There are lots of conventions or rules-of-thumb in the survey design business. Here's a checklist
of some of the most important items. You can use this checklist to review your instrument:
Top of Form

start with easy, nonthreatening questions

put more difficult, threatening questions near end

never start a mail survey with an open-ended question

for historical demographics, follow chronological order

ask about one topic at a time

when switching topics, use a transition

reduce response set (the tendency of respondent to just keep checking the same response)

for filter or contingency questions, make a flowchart

Bottom of Form

The Golden Rule

You are imposing in the life of your respondent. You are asking for their time, their attention,
their trust, and often, for personal information. Therefore, you should always keep in mind the
"golden rule" of survey research (and, I hope, for the rest of your life as well!):
Do unto your respondents as you would have them do unto
you!
To put this in more practical terms, you should keep the following in mind:
• Thank the respondent at the beginning for allowing you to conduct your study
• Keep your survey as short as possible -- only include what is absolutely necessary
• Be sensitive to the needs of the respondent
• Be alert for any sign that the respondent is uncomfortable
• Thank the respondent at the end for participating
• Assure the respondent that you will send a copy of the final results

Interviews
Interviews are among the most challenging and rewarding forms of measurement. They require a
personal sensitivity and adaptability as well as the ability to stay within the bounds of the
designed protocol. Here, I describe the preparation you need to do for an interview study and the
process of conducting the interview itself.
Preparation
The Role of the Interviewer
The interviewer is really the "jack-of-all-trades" in survey research. The interviewer's role is
complex and multifaceted. It includes the following tasks:
• Locate and enlist cooperation of respondents
The interviewer has to find the respondent. In door-to-door surveys, this means being able to
locate specific addresses. Often, the interviewer has to work at the least desirable times (like
immediately after dinner or on weekends) because that's when respondents are most readily
available.
• Motivate respondents to do good job
If the interviewer does not take the work seriously, why would the respondent? The interviewer
has to be motivated and has to be able to communicate that motivation to the respondent. Often,
this means that the interviewer has to be convinced of the importance of the research.
• Clarify any confusion/concerns
Interviewers have to be able to think on their feet. Respondents may raise objections or concerns
that were not anticipated. The interviewer has to be able to respond candidly and informatively.
• Observe quality of responses
Whether the interview is personal or over the phone, the interviewer is in the best position to
judge the quality of the information that is being received. Even a verbatim transcript will not
adequately convey how seriously the respondent took the task, or any gestures or body language
that were evident.
• Conduct a good interview
Last, and certainly not least, the interviewer has to conduct a good interview! Every interview
has a life of its own. Some respondents are motivated and attentive, others are distracted or
disinterested. The interviewer also has good or bad days. Assuring a consistently high-quality
interview is a challenge that requires constant effort.
Training the Interviewers
One of the most important aspects of any interview study is the training of the interviewers
themselves. In many ways the interviewers are your measures, and the quality of the results is
totally in their hands. Even in small studies involving only a single researcher-interviewer, it is
important to organize in detail and rehearse the interviewing process before beginning the formal
study.
Here are some of the major topics that should be included in interviewer training:
• Describe the entire study
Interviewers need to know more than simply how to conduct the interview itself. They should
learn about the background for the study, previous work that has been done, and why the study is
important.
• State who is sponsor of research
Interviewers need to know who they are working for. They -- and their respondents -- have a
right to know not just what agency or company is conducting the research, but also, who is
paying for the research.
• Teach enough about survey research
While you seldom have the time to teach a full course on survey research methods, the
interviewers need to know enough that they respect the survey method and are motivated.
Sometimes it may not be apparent why a question or set of questions was asked in a particular
way. The interviewers will need to understand the rationale for how the instrument was
constructed.
• Explain the sampling logic and process
Naive interviewers may not understand why sampling is so important. They may wonder why
you go through all the difficulties of selecting the sample so carefully. You will have to explain
that sampling is the basis for the conclusions that will be reached and for the degree to which
your study will be useful.
• Explain interviewer bias
Interviewers need to know the many ways that they can inadvertently bias the results. And, they
need to understand why it is important that they not bias the study. This is especially a problem
when you are investigating political or moral issues on which people have strongly held
convictions. While the interviewer may think they are doing good for society by slanting results
in favor of what they believe, they need to recognize that doing so could jeopardize the entire
study in the eyes of others.
• "Walk through" the interview
When you first introduce the interview, it's a good idea to walk through the entire protocol so the
interviewers can get an idea of the various parts or phases and how they interrelate.
• Explain respondent selection procedures, including
• reading maps
It's astonishing how many adults don't know how to follow directions on a map. In personal
interviews, the interviewer may need to locate respondents who are spread over a wide
geographic area. And, they often have to navigate by night (respondents tend to be most
available in evening hours) in neighborhoods they're not familiar with. Teaching basic map
reading skills and confirming that the interviewers can follow maps is essential.
• identifying households
In many studies it is impossible in advance to say whether every sample household meets the
sampling requirements for the study. In your study, you may want to interview only people who
live in single family homes. It may be impossible to distinguish townhouses and apartment
buildings in your sampling frame. The interviewer must know how to identify the appropriate
target household.
• identify respondents
Just as with households, many studies require respondents who meet specific criteria. For
instance, your study may require that you speak with a male head-of-household between the ages
of 30 and 40 who has children under 18 living in the same household. It may be impossible to
obtain statistics in advance to target such respondents. The interviewer may have to ask a series
of filtering questions before determining whether the respondent meets the sampling needs.
• Rehearse interview
You should probably have several rehearsal sessions with the interviewer team. You might even
videotape rehearsal interviews to discuss how the trainees responded in difficult situations. The
interviewers should be very familiar with the entire interview before ever facing a respondent.
• Explain supervision
In most interview studies, the interviewers will work under the direction of a supervisor. In some
contexts, the supervisor may be a faculty advisor; in others, they may be the "boss." In order to
assure the quality of the responses, the supervisor may have to observe a subsample of
interviews, listen in on phone interviews, or conduct follow-up assessments of interviews with
the respondents. This can be very threatening to the interviewers. You need to develop an
atmosphere where everyone on the research team -- interviewers and supervisors -- feel like
they're working together towards a common end.
• Explain scheduling
The interviewers have to understand the demands being made on their schedules and why these
are important to the study. In some studies it will be imperative to conduct the entire set of
interviews within a certain time period. In most studies, it's important to have the interviewers
available when it's convenient for the respondents, not necessarily the interviewer.
The Interviewer's Kit
It's important that interviewers have all of the materials they need to do a professional job.
Usually, you will want to assemble an interviewer kit that can be easily carried and includes all
of the important materials such as:
• a "professional-looking" 3-ring notebook (this might even have the logo of the company
or organization conducting the interviews)
• maps
• sufficient copies of the survey instrument
• official identification (preferable a picture ID)
• a cover letter from the Principal Investigator or Sponsor
 • a phone number the respondent can call to verify the interviewer's authenticity
The Interview
So all the preparation is complete, the training done, the interviewers ready to proceed, their
"kits" in hand. It's finally time to do an actual interview. Each interview is unique, like a small
work of art (and sometimes the art may not be very good). Each interview has its own ebb and
flow -- its own pace. To the outsider, an interview looks like a fairly standard, simple, prosaic
effort. But to the interviewer, it can be filled with special nuances and interpretations that aren't
often immediately apparent. Every interview includes some common components. There's the
opening, where the interviewer gains entry and establishes the rapport and tone for what follows.
There's the middle game, the heart of the process, that consists of the protocol of questions and
the improvisations of the probe. And finally, there's the endgame, the wrap-up, where the
interviewer and respondent establish a sense of closure. Whether it's a two-minute phone
interview or a personal interview that spans hours, the interview is a bit of theater, a mini-drama
that involves real lives in real time.
Opening Remarks
In many ways, the interviewer has the same initial problem that a salesperson has. You have to
get the respondent's attention initially for a long enough period that you can sell them on the idea
of participating in the study. Many of the remarks here assume an interview that is being
conducted at a respondent's residence. But the analogies to other interview contexts should be
straightforward.
• Gaining entry
The first thing the interviewer must do is gain entry. Several factors can enhance the prospects.
Probably the most important factor is your initial appearance. The interviewer needs to dress
professionally and in a manner that will be comfortable to the respondent. In some contexts a
business suit and briefcase may be appropriate. In others, it may intimidate. The way the
interviewer appears initially to the respondent has to communicate some simple messages -- that
you're trustworthy, honest, and non-threatening. Cultivating a manner of professional
confidence, the sense that the respondent has nothing to worry about because you know what
you're doing -- is a difficult skill to teach and an indispensable skill for achieving initial entry.
• Doorstep technique
You're standing on the doorstep and someone has opened the door, even if only halfway. You
need to smile. You need to be brief. State why you are there and suggest what you would like the
respondent to do. Don't ask -- suggest what you want. Instead of saying "May I come in to do an
interview?", you might try a more imperative approach like " I'd like to take a few minutes of
your time to interview you for a very important study."
• Introduction
If you've gotten this far without having the door slammed in your face, chances are you will be
able to get an interview. Without waiting for the respondent to ask questions, you should move to
introducing yourself. You should have this part of the process memorized so you can deliver the
essential information in 20-30 seconds at most. State your name and the name of the organization
you represent. Show your identification badge and the letter that introduces you. You want to
have as legitimate an appearance as possible. If you have a three-ring binder or clipboard with
the logo of your organization, you should have it out and visible. You should assume that the
respondent will be interested in participating in your important study -- assume that you will be
doing an interview here.
• Explaining the study
At this point, you've been invited to come in (After all, you're standing there in the cold, holding
an assortment of materials, clearly displaying your credentials, and offering the respondent the
chance to participate in an interview -- to many respondents, it's a rare and exciting event. They
hardly ever get asked their views about anything, and yet they know that important decisions are
made all the time based on input from others.). Or, the respondent has continued to listen long
enough that you need to move onto explaining the study. There are three rules to this critical
explanation: 1) Keep it short; 2) Keep it short; and 3) Keep it short! The respondent doesn't have
to or want to know all of the neat nuances of this study, how it came about, how you convinced
your thesis committee to buy into it, and so on. You should have a one or two sentence
description of the study memorized. No big words. No jargon. No detail. There will be more than
enough time for that later (and you should bring some written materials you can leave at the end
for that purpose). This is the "25 words or less" description. What you should spend some time
on is assuring the respondent that you are interviewing them confidentially, and that their
participation is voluntary.
Asking the Questions
You've gotten in. The respondent has asked you to sit down and make yourself comfortable. It
may be that the respondent was in the middle of doing something when you arrived and you may
need to allow them a few minutes to finish the phone call or send the kids off to do homework.
Now, you're ready to begin the interview itself.
• Use questionnaire carefully, but informally
The questionnaire is your friend. It was developed with a lot of care and thoughtfulness. While
you have to be ready to adapt to the needs of the setting, your first instinct should always be to
trust the instrument that was designed. But you also need to establish a rapport with the
respondent. If you have your face in the instrument and you read the questions, you'll appear
unprofessional and disinterested. Even though you may be nervous, you need to recognize that
your respondent is most likely even more nervous. If you memorize the first few questions, you
can refer to the instrument only occasionally, using eye contact and a confident manner to set the
tone for the interview and help the respondent get comfortable.
• Ask questions exactly as written
Sometimes an interviewer will think that they could improve on the tone of a question by altering
a few words to make it simpler or more "friendly." DON'T. You should ask the questions as they
are on the instrument. If you had a problem with a question, the time to raise it was during the
training and rehearsals, not during the actual interview. It is important that the interview be as
standardized as possible across respondents (this is true except in certain types of exploratory or
interpretivist research where the explicit goal is to avoid any standardizing). You may think the
change you made was inconsequential when, in fact, it may change the entire meaning of the
question or response.
• Follow the order given
Once you know an interview well, you may see a respondent bring up a topic that you know will
come up later in the interview. You may be tempted to jump to that section of the interview
while you're on the topic. DON'T. You are more likely to lose your place. You may omit
questions that build a foundation for later questions.
• Ask every question
Sometimes you'll be tempted to omit a question because you thought you already heard what the
respondent will say. Don't assume that. For example, let's say you were conducting an interview
with college age women about the topic of date rape. In an earlier question, the respondent
mentioned that she knew of a woman on her dormitory floor who had been raped on a date
within the past year. A few questions later, you are supposed to ask "Do you know of anyone
personally who was raped on a date?" You figure you already know that the answer is yes, so
you decide to skip the question. Instead, you might say something like "I know you may have
already mentioned this, but do you know of anyone personally who was raped on a date?" At this
point, the respondent may say something like "Well, in addition to the woman who lived down
the hall in my dorm, I know of a friend from high school who experienced date rape." If you
hadn't asked the question, you would never have discovered this detail.
• Don't finish sentences
I don't know about you, but I'm one of those people who just hates to be left hanging. I like to
keep a conversation moving. Once I know where a sentence seems to be heading, I'm aching to
get to the next sentence. I finish people's sentences all the time. If you're like me, you should
practice the art of patience (and silence) before doing any interviewing. As you'll see below,
silence is one of the most effective devices for encouraging a respondent to talk. If you finish
their sentence for them, you imply that what they had to say is transparent or obvious, or that you
don't want to give them the time to express themselves in their own language.
Obtaining Adequate Responses - The Probe
OK, you've asked a question. The respondent gives a brief, cursory answer. How do you elicit a
more thoughtful, thorough response? You probe.
• Silent probe
The most effective way to encourage someone to elaborate is to do nothing at all - just pause and
wait. This is referred to as the "silent" probe. It works (at least in certain cultures) because the
respondent is uncomfortable with pauses or silence. It suggests to the respondent that you are
waiting, listening for what they will say next.
• Overt encouragement
At times, you can encourage the respondent directly. Try to do so in a way that does not imply
approval or disapproval of what they said (that could bias their subsequent results). Overt
encouragement could be as simple as saying "Uh-huh" or "OK" after the respondent completes a
thought.
• Elaboration
You can encourage more information by asking for elaboration. For instance, it is appropriate to
ask questions like "Would you like to elaborate on that?" or "Is there anything else you would
like to add?"
• Ask for clarification
Sometimes, you can elicit greater detail by asking the respondent to clarify something that was
said earlier. You might say, "A minute ago you were talking about the experience you had in
high school. Could you tell me more about that?"
• Repetition
This is the old psychotherapist trick. You say something without really saying anything new. For
instance, the respondent just described a traumatic experience they had in childhood. You might
say "What I'm hearing you say is that you found that experience very traumatic." Then, you
should pause. The respondent is likely to say something like "Well, yes, and it affected the rest
of my family as well. In fact, my younger sister..."
Recording the Response
Although we have the capability to record a respondent in audio and/or video, most interview
methodologists don't think it's a good idea. Respondents are often uncomfortable when they
know their remarks will be recorded word-for-word. They may strain to only say things in a
socially acceptable way. Although you would get a more detailed and accurate record, it is likely
to be distorted by the very process of obtaining it. This may be more of a problem in some
situations than in others. It is increasingly common to be told that your conversation may be
recorded during a phone interview. And most focus group methodologies use unobtrusive
recording equipment to capture what's being said. But, in general, personal interviews are still
best when recorded by the interviewer using pen and paper. Here, I assume the paper-and-pencil
approach.
• Record responses immediately
The interviewer should record responses as they are being stated. This conveys the idea that you
are interested enough in what the respondent is saying to write it down. You don't have to write
down every single word -- you're not taking stenography. But you may want to record certain
key phrases or quotes verbatim. You need to develop a system for distinguishing what the
respondent says verbatim from what you are characterizing (how about quotations, for instance!).
• Include all probes
You need to indicate every single probe that you use. Develop a shorthand for different standard
probes. Use a clear form for writing them in (e.g., place probes in the left margin).
• Use abbreviations where possible
Abbreviations will help you to capture more of the discussion. Develop a standardized system
(e.g., R=respondent; DK=don't know). If you create an abbreviation on the fly, have a way of
indicating its origin. For instance, if you decide to abbreviate Spouse with an 'S', you might make
a notation in the right margin saying "S=Spouse."
Concluding the Interview
When you've gone through the entire interview, you need to bring the interview to closure. Some
important things to remember:
• Thank the respondent
Don't forget to do this. Even if the respondent was troublesome or uninformative, it is important
for you to be polite and thank them for their time.
• Tell them when you expect to send results
I hate it when people conduct interviews and then don't send results and summaries to the people
who they get the information from. You owe it to your respondent to show them what you
learned. Now, they may not want your entire 300-page dissertation. It's common practice to
prepare a short, readable, jargon-free summary of interviews that you can send to the
respondents.
• Don't be brusque or hasty
Allow for a few minutes of winding down conversation. The respondent may want to know a
little bit about you or how much you like doing this kind of work. They may be interested in how
the results will be used. Use these kinds of interests as a way to wrap up the conversation. As
you're putting away your materials and packing up to go, engage the respondent. You don't want
the respondent to feel as though you completed the interview and then rushed out on them -- they
may wonder what they said that was wrong. On the other hand, you have to be careful here.
Some respondents may want to keep on talking long after the interview is over. You have to find
a way to politely cut off the conversation and make your exit.
• Immediately after leaving -- write down any notes about how the interview went
Sometimes you will have observations about the interview that you didn't want to write down
while you were with the respondent. You may have noticed them get upset at a question, or you
may have detected hostility in a response. Immediately after the interview you should go over
your notes and make any other comments and observations -- but be sure to distinguish these
from the notes made during the interview (you might use a different color pen, for instance).

Plus & Minus of Survey Methods

It's hard to compare the advantages and disadvantages of the major different survey types. Even
though each type has some general advantages and disadvantages, there are exceptions to almost
every rule. Here's my general assessment. Perhaps you would differ in your ratings here or there,
but I think you'll generally agree.

Issue Questionnaire Interview

Drop-
Group Mail Personal Phone
Off

Are Visual Presentations Possible? Yes Yes Yes Yes No

Are Long Response Categories

Yes Yes Yes ??? No
Possible?

Is Privacy A Feature? No Yes No Yes ???

Is the Method Flexible? No No No Yes Yes

Are Open-ended Questions Feasible? No No No Yes Yes

Is Reading & Writing Needed? ??? Yes Yes No No

Can You Judge Quality of Response? Yes No ??? Yes ???

Are High Response Rates Likely? Yes No Yes Yes No

Can You Explain Study in Person? Yes No Yes Yes ???

Is It Low Cost? Yes Yes No No No

Are Staff & Facilities Needs Low? Yes Yes No No No

Does It Give Access to Dispersed

No Yes No No No
Samples?

Does Respondent Have Time to

No Yes Yes No No
Formulate Answers?

Is There Personal Contact? Yes No Yes Yes No

Is A Long Survey Feasible? No No No Yes No

Is There Quick Turnaround? No Yes No No Yes

Scaling
Scaling is the branch of measurement that involves the construction of an instrument that
associates qualitative constructs with quantitative metric units. Scaling evolved out of efforts in
psychology and education to measure "unmeasurable" constructs like authoritarianism and self
esteem. In many ways, scaling remains one of the most arcane and misunderstood aspects of
social research measurement. And, it attempts to do one of the most difficult of research tasks --
measure abstract concepts.
Most people don't even understand what scaling is. The basic idea of scaling is described in
General Issues in Scaling, including the important distinction between a scale and a response
format. Scales are generally divided into two broad categories: unidimensional and
multidimensional. The unidimensional scaling methods were developed in the first half of the
twentieth century and are generally named after their inventor. We'll look at three types of
unidimensional scaling methods here:
• Thurstone or Equal-Appearing Interval Scaling
• Likert or "Summative" Scaling
• Guttman or "Cumulative" Scaling
In the late 1950s and early 1960s, measurement theorists developed more advanced techniques
for creating multidimensional scales. Although these techniques are not considered here, you
may want to look at the method of concept mapping that relies on that approach to see the power
of these multivariate methods.
Bottom of Form