Designing Multimethod Research

Oxford Handbooks Online

Designing Multimethod Research
Albert Hunter and John D. Brewer
The Oxford Handbook of Multimethod and Mixed Methods Research Inquiry
Edited by Sharlene Nagy Hesse-Biber and R. Burke Johnson

Print Publication Date: Aug 2015

Subject: Psychology, Psychological Methods and Measurement
Online Publication Date: Jan 2016 DOI: 10.1093/oxfordhb/9780199933624.013.13

Abstract and Keywords

The chapter begins with the “spirit of multimethod research,” which refers to an
openness to serendipity and humble recognition that all methods have strengths and
weaknesses and that by combining different methods one has compensating strengths
leading to more credible results in the face of a series of skeptics’ questions.
“Multimethods” involves combining any different methods while” mixed methods” more
specifically focuses on combining qualitative and quantitative methods. Typologies of
multimethod research from the literature based on characteristics and sequencing of
methods themselves are briefly explored. The focus of the chapter is on the design of
multimethod research in different subfields elucidating early historical exemplars and
current developments. The selective and nonexhaustive subfields include words and
deeds, rational actor, contextual effects, biomedical-social, evaluation, historical/
comparative, and disaster research. Much of the chapter uses a positivist framework but
concludes with a discussion of how multimethods are relevant for postpositive rhetorical
and narrative frames.

Keywords: multimethod research, compensating strengths and weaknesses, skeptics' questions, exemplars, post-
positivist, rhetorical, narrative frames, design, multimethod, mixed method, historical exemplars

Preamble: The Spirit of Multimethod Research

In addressing the question of research design we do not provide a “cookbook” or
textbook normative approach to design where all is codified in a sequence of steps such
that by doing A then B and combining with C one will produce D, an optimal outcome of
research findings. We would rather stress the “art” of multimethod research—recognizing
that good science exists to the degree it addresses a limited but profound set of

Page 1 of 36

PRINTED FROM OXFORD HANDBOOKS ONLINE (www.oxfordhandbooks.com). © Oxford University Press, 2018. All Rights
Reserved. Under the terms of the licence agreement, an individual user may print out a PDF of a single chapter of a title in
Oxford Handbooks Online for personal use (for details see Privacy Policy and Legal Notice).

Subscriber: Goldsmiths College, University of London; date: 02 February 2019

Designing Multimethod Research

questions. We have referred to these as the skeptics’ questions: an overarching concern

with content, not a fetishsized concern with form—what some have labelled scientism.

We must not have an overreified notion of what science is but rather a more realistic
empirical understanding of how science is actually done. As repeated studies in the
sociology of science, or science studies suggest, it is better to pay attention to what
practitioners say they really did, not what the methods textbooks say one should do
(Denzin, 1970). As Paul Lazarsfeld (1945), an early practitioner of multimethod as well as
survey research, stressed, “A good methodological investigation is not concerned with
what a science should be: it tries to clarify what a science is and how it obtains its
results” (p. vii; see also Jahoda, Lazarsfeld, & Zeisel, 1933/1971).

If one is to learn by studying then perhaps it is best to study exemplars, reports on

research itself, not rules. This same spirit we suggest may carry over into the teaching
and mentoring of science education. In the hard sciences (physics, chemistry, biology,
zoology) one does not have courses in methods per se; rather one acquires methods
through the (p. 186) older apprentice model of working in labs, and as a research assistant
one learns methods by doing research, not reading about how one should do research.
This is a distinction we will see shortly is akin to distinguishing between “saying versus
doing” or “words versus deeds.” Methods are what one does—not to say there are no
limits and boundaries, but these are few and easily communicated as ideals—reliability
and reproducibility, openness, no fabrication or fraud, and so on. Beyond that,
researchers should show their data, explain how they obtained their data, how they
analyzed their data, and consider alternative interpretations, summarizations, and
generalizations as well as limitations and threats to validity. The objective is to build an
argument that is consistent (logical), corresponding (to the data), and convincing
(believable).

The Skeptics’ Questions

Research methods, in short, must address a few very basic questions true of all scientific
investigations, and these normative ideals ask that research be systematic, open,
reproducible, reliable, valid, and believable after scrutiny with a healthy skepticism.

Elsewhere we have enumerated some of these basic skeptics’ questions as follows:

(1) Absolute theoretical adequacy—to what degree does a theory explain the full set
of empirical findings?
(2) Relative theoretical adequacy—how well does a theory explain data compared to
other possible theories?
(3) Questions of methodological bias—how might the data to be explained be
influenced by the particular method(s) used to obtain the data?
(4) Questions of measurement and conceptualization—how well have the concepts of
a theory been measured?

Page 2 of 36

PRINTED FROM OXFORD HANDBOOKS ONLINE (www.oxfordhandbooks.com). © Oxford University Press, 2018. All Rights
Reserved. Under the terms of the licence agreement, an individual user may print out a PDF of a single chapter of a title in
Oxford Handbooks Online for personal use (for details see Privacy Policy and Legal Notice).

Subscriber: Goldsmiths College, University of London; date: 02 February 2019

Designing Multimethod Research

(5) Questions of causal inference—how well have research methods addressed the
cause-and-effect relationships of a theory’s hypotheses
(6) Questions of generalizability—how generalizable are the findings of research
given the methods used?
(7) Questions of realism versus simplification—how much realism versus
simplification do the research’s methods provide? (Brewer & Hunter, 1989, 2006)

All research, whatever the method, should address these questions, and different
methods may address one or another of the questions better than others. The
multimethods we suggest may be particularly helpful in addressing more of them, thereby
reducing the skepticism about the research results or, conversely, increasing their
credibility.

The overarching purpose of our conceiving and writing Multimethod Research (Brewer &
Hunter, 1989) over some three decades ago was to overcome the continuing and at times
acrimonious debates between practitioners of different techniques of research, especially
but not exclusively between quantitative and qualitative approaches in the social
sciences. It was a disciplinary concern as much as a narrow methodological one—with an
overall objective of reasserting the centrality of the old Baconian generic goal of an
eclectic but rigorous method of science itself.

The earlier debates were useful for self-reflection and continuing refinement of different
techniques highlighting the limitations noted by critics as well as the strengths asserted
by proponents often in hyperbolic rhetoric on both sides. Our goal was to discipline but
not stifle the debates and to bring about a tolerance and mutual understanding of a
unified endeavor—advancing the singular overarching method of science itself.

What Do We Mean By “Design”?

Design means (for us) a number of different things. Design may, on the one hand, be the
outcome of prescriptive planning as in research proposals where specification and
selection of such elements as units of investigation, universes, sampling frames, sampling
techniques, measurement, data transformation, and modes of analysis are thoughtfully
defined in detail. On the other hand design may be discerned in post hoc pattern
recognition of what has been an unfolding, evolutionary, pragmatic adaptation in the
research process. In short, design can consciously occur before starting the research, or
the process can occur as the research unfolds in an organic, opportunistic, and
serendipitous fashion. We try to encompass both of these meanings—the enacted and the
emergent—in our discussion of design. We suggest that most research progresses as a
combination of some preplanning coupled with judicious emergent decision-making as
the research is carried out. We would encourage a spirit of a competently trained
pragmatic openness to serendipity; in short, the idea of “design” in designing
multimethod research is more of a Darwinian notion of evolution and adaptation rather
than an overreliance on an authoritative “Intelligent Designer.”

Page 3 of 36

PRINTED FROM OXFORD HANDBOOKS ONLINE (www.oxfordhandbooks.com). © Oxford University Press, 2018. All Rights
Reserved. Under the terms of the licence agreement, an individual user may print out a PDF of a single chapter of a title in
Oxford Handbooks Online for personal use (for details see Privacy Policy and Legal Notice).

Subscriber: Goldsmiths College, University of London; date: 02 February 2019

Designing Multimethod Research

The evolution of multimethod designs, as with all methods, can occur at two levels
(p. 187)

—at the micro level with the opportunistic unfolding and combining of specific techniques
of data collection and analysis present in a single research project, and at the macro level
of a discipline as a whole as Mario Small (2011) demonstrated in his comprehensive
review of multimethod research in sociology as he traced the unfolding natural history of
specific combinations of different types of methods over recent decades. The multimethod
spirit is one that is open to new, innovative, and at times unanticipated techniques.

Prepared Contingency
One way individuals learn different methods is through the study of prior exemplars and
also through an apprenticeship in mentoring relationships during the actual conduct of
research itself. An exposure to, understanding of, and capacity to use different methods is
an ongoing learning experience throughout one’s career, a learning process greatly aided
by an openness to a multimethod perspective. Some individuals may develop varying
degrees of mastery of a variety of techniques, say for data collection or data analysis,
while others may become highly specialized in this or that singular technique. The former
person may craft a multimethod piece of research on his or her own, while the latter may
still participate in multimethod research through a division of labor within a collective
research project. Such was the case for example with the Reactions to Crime Project
carried out by Northwestern University’s Center for Urban Affairs and Policy Research in
the 1970s and 1980s (Hunter & Maxfield, 1980). But, to work, the collective multimethod
research project must have a collective spirit that recognizes the strengths and
limitations of the different methods, a spirit that does not just tolerate but emphasizes
and appreciates the need for rigor and quality in the application of different methods in
the overall scientific enterprise. In sum, multimethod research does not disparage
monomethod research and in fact encourages more reflexive and refined sophistication in
the mastering of each method with the goal of enacting the highest quality competent
research. At the same time, the multimethod spirit encourages a “design for serendipity”
recognizing that opportunistic and creative insights come to the well-prepared and
reflective mind.

Methods Are What Researchers Do

Case studies in the ethnography of science have shown repeatedly that actual research is
different than any ideal, and, we suggest, this is true of multimethod research as well.
The oft told story of W. I. Thomas (1923) discovering a packet of letters thrown out in an
alley and seeing them as a potential data source for studying Polish peasants who had
immigrated to America is perhaps apocryphal but underscores the need for openness to
diverse data and methods. We are not interested in prescription or preaching with
respect to multimethod research. It is an approach we highly recommend when and
where appropriate. Monomethod research is at times under certain conditions and
contexts the way to go. We definitely prefer well-executed monomethod research to less

Page 4 of 36

PRINTED FROM OXFORD HANDBOOKS ONLINE (www.oxfordhandbooks.com). © Oxford University Press, 2018. All Rights
Reserved. Under the terms of the licence agreement, an individual user may print out a PDF of a single chapter of a title in
Oxford Handbooks Online for personal use (for details see Privacy Policy and Legal Notice).

Subscriber: Goldsmiths College, University of London; date: 02 February 2019

Designing Multimethod Research

competent multimethod research. The use of multimethods alone does not exonerate nor
should it be a cloak for incompetence.

Multimethod Research Design

Multimethod research may be broadly defined as the practice of employing two or more
different methods or styles of research within the same study or research program rather
than confining the research to the use of a single method (Brewer & Hunter, 1989, 2006).
Unlike mixed method research, it is not restricted to combining qualitative and
quantitative methods but rather is open to the full variety of possible methodological
combinations.

As there are numerous methods available to social researchers, and a variety of ways in
which these methods can be combined, the topic of multimethod research design involves
foremost asking (a) which methods are combined with which other methods and (b) how
are the different methods deployed and implemented in relation to one another in the
research process.

Typologies of Mixing
Prior commentators on multimethod research have produced a number of typologies
reflecting the “what” and “how” of combining or mixing different distinct methods. We
will briefly explore a few of these as developed by others. We note that most commonly in
the literature “mixed methods” is used to refer to mixing quantitative and a qualitative
method, while “multimethod” more broadly refers to mixing of two or more methods—
regardless of whether they are qualitative or quantitative. In short, mixed method is a
subset of multimethod.

One of the more formal and systematic typologies is that by Janice Morse (2003) who
began with (p. 188) the basic qualitative quantitative distinction that she equated with a
deductive and inductive drive, respectively. She then paired each with either a second
quantitative or qualitative method and then subdivided each of those four cases in turn
into a simultaneous or sequential timing of the deployment of the methods. The result
was an eight-fold typology that she then analyzed in more depth as to their contribution
to theory.

Johnson and Turner (2003) derived a typology based on the principal method of data
collection by first distinguishing among the methods of questionnaires, interviews, focus
groups, tests, observation, and secondary data. They then considered that each of these
methods may in and of itself use mixed data collection strategies that could include two
quantitative, two qualitative, or one qualitative and one quantitative. The first two
“mixes” would be multimethod, while the third would be defined as an example of “mixed
methods.” Finally, they suggested the six different methods themselves may be combined

Page 5 of 36

PRINTED FROM OXFORD HANDBOOKS ONLINE (www.oxfordhandbooks.com). © Oxford University Press, 2018. All Rights
Reserved. Under the terms of the licence agreement, an individual user may print out a PDF of a single chapter of a title in
Oxford Handbooks Online for personal use (for details see Privacy Policy and Legal Notice).

Subscriber: Goldsmiths College, University of London; date: 02 February 2019

Designing Multimethod Research

to create a “continuum” of a variety of methods and varied qualitative/quantitative

combinations.

We will look at two other typologies that are more inductive in their derivation by Jerry
Jacobs (2005) and Mario Small (2011), respectively. Let us begin by examining research
published recently by Jerry Jacobs in the American Sociological Review, the official
journal of the American Sociological Association. This seems a promising way to lay out
the basic dimensions of multimethod and mixed methods research design as now
practiced in the discipline with which we are most familiar. Jacobs has helpfully published
a brief survey of the papers published between 2003 and 2005, a period when he was
editor of the American Sociological Review. He reported that a quarter of the papers he
accepted for publication (17 out of 65) involved multiple method research, by which he
means studies that “draw on data from more than one source and present more than one
type of analysis” (Jacobs, 2005).

Jacobs (2005) found several different combinations of methods in this group of studies.
He noted that these multiple method studies “often but not always combined quantitative
and qualitative data.” First, statistical surveys were combined with qualitative interviews.
The interviews might either precede the survey to refine its questions or be conducted
later to increase understanding and deepen interpretation of the survey data. Second,
there were multiple quantitative approaches: for example, a field experiment followed
later by a survey of the same subjects to test and compare with the experimental results;
analysis of survey data from 30 countries to bolster a hypothesis derived from aggregate
data from those countries. Finally, historical analyses employed qualitative and
quantitative analyses of historical documents and archival data, and quantitative analysis
of survey data were available as well as qualitative interview data from still-living
participants. These different sorts of data were sometimes used concurrently to provide a
more complete picture and sometimes in sequence to form and test hypotheses.

To put Jacobs’s (2005) report in perspective we have examined two other samples of the
American Sociological Review, one roughly 20 years earlier (1984 and 1985) and the
second from 2011 to the present. A striking contrast in these different time periods is
that while Jacobs identified 17 multiple method studies in the 2003–2005 period, there
were 10 such studies during 1984 and 1985, and 32 during 2011 and 2012. Clearly there
is a linear trend of increasing publication of multimethod studies within sociology.

Mario Small (2011) in an Annual Review of Sociology article similarly developed an

inductive typology based on an extensive review of the sociological literature. He too first
noted the burgeoning number of studies that have come to use multiple methods in
recent decades. He also noted that the usual combined qualitative/quantitative meaning
of the category is unnecessarily restrictive as different qualitative methods and different
quantitative methods may also be combined, and that furthermore the distinction of what
constitutes quantitative and qualitative is itself a “fuzzy” distinction.

Page 6 of 36

PRINTED FROM OXFORD HANDBOOKS ONLINE (www.oxfordhandbooks.com). © Oxford University Press, 2018. All Rights
Reserved. Under the terms of the licence agreement, an individual user may print out a PDF of a single chapter of a title in
Oxford Handbooks Online for personal use (for details see Privacy Policy and Legal Notice).

Subscriber: Goldsmiths College, University of London; date: 02 February 2019

Designing Multimethod Research

He began his typology by distinguishing between data collection and data analysis,
recognizing, for example, that one could have a qualitative analysis of quantitative data
and conversely a quantitative analysis of qualitative data. He then proceeded to elucidate
examples of these varying types. He also considered the epistemology of multimethod
research, which he noted is heavily rooted in pragmatism, and the primary motivations of
researchers, which are either confirmation or complementarity. He further explored
questions of sequencing and nesting in data collection and cross-over and integration in
data analysis. Finally he noted the enduring questions of commensurability and
specialization among methods. As he concluded:

the challenges of mixed methods research reflect those of sociology writ large, a
discipline whose core methodological pluralism has produced, over (p. 189) its
history, periods of conflict and cooperation, but few of lasting resolution. . . .
Mixed method projects provide both the challenge and the opportunity for
researchers to resolve some of the ambiguities that result from pluralism. (Small,
2011, p. 79)

An Inductive Typology of Fruitful Substantive

Arenas for Multimethod Social Science
Research
The previous discussion of multimethod research highlights typologies of different
combinations of methods that are derived from characteristics of the methods themselves
—such as the basic quantitative qualitative distinction, or relative significance as to which
method has primacy in the research versus which plays a supportive role, or the mere
sequencing and timing as to which comes first and which follows. These typologies are
valuable for giving some coherence and organization to the rapidly expanding
accumulation of multimethod studies. One must be careful, however, to not overreify
these typologies and potentially advance them as normatively prescriptive strategies for
conducting multimethod research. All research, multimethod included, is, we contend, a
combination of science and art, design and serendipity, thoughtful planning and
pragmatic opportunism.

We use here a more inductive approach following the lead of Jacobs (2005) and Small
(2011) and develop a typology of multimethod research that begins not from first
principles about methods but from different generalized types of substantive questions
that have already been explored in the literature using multimethod research. The types
of substantive questions are varied, and some that we identify, like that exploring the
differences between words and deeds, have a long history in the literature, while others,
such as combing biomedical markers and interview data, are relatively recent in origin.
We present this typology as neither exhaustive nor necessarily mutually exclusive and do
so in the hopes of presenting categories that will be useful in organizing and bringing
some order to the field. But, in contrast to most multimethod typologies, we offer these
Page 7 of 36

PRINTED FROM OXFORD HANDBOOKS ONLINE (www.oxfordhandbooks.com). © Oxford University Press, 2018. All Rights
Reserved. Under the terms of the licence agreement, an individual user may print out a PDF of a single chapter of a title in
Oxford Handbooks Online for personal use (for details see Privacy Policy and Legal Notice).

Subscriber: Goldsmiths College, University of London; date: 02 February 2019

Designing Multimethod Research

less as prescriptive normative guides and more as sensitizing exemplars. As one prepares
to conduct a piece of research and thinks about how to possibly incorporate a
multimethod style, one can ask substantively, not just methodologically: What kind of
research question is this? In such a case, one is being more true to the old adage that
research questions should be driven by theoretical substance not methodological
proclivities. Substance should dictate methods, not the reverse. This is an admonition
shared widely across science in general, not just the social sciences.

For example, a recent article in Science (Joppa et al., 2013) reporting a survey study of
400 biologists using canned modeling programs in the substantive area of “species
distribution modeling” quotes one scientist: “The research question and the data should
be king, with an approach being selected on the basis that it is appropriate to both the
research question and the data rather than the research question and the data being
selected to fit the approach which a person knows how to use.” While another biologist
echoed a multimethod thesis: “We don’t need fancier software, we need people who
understand ecology and the importance of multiple types of data. . . The key is the ability
to think in ecological terms”(p. 815).

Substantive Arenas
The following are a few of these types of substantive arenas often using multimethod
research that we have begun to explore and that we briefly elaborate in the following
discussion:

(1) Words and Deeds

(2) Rational Action
(3) Contextual Effects
(4) Biomedical Social Research
(5) Evaluation Research
(6) Historical/Comparative
(7) Disaster Research

Words and Deeds

Words are what people say and are windows into what they think, while deeds are what
people do—their behavior. We are most interested in the latter, because that is what has
direct consequences for us and that is what we are most interested in anticipating so we
may adapt our own behavior accordingly.

We acknowledge that words may be thought of as a form of behavior, so in this light we

are comparing two different forms of behavior with one another. The words, which give
the participants’ “meanings,” are closer to thoughts and ideas, or, using a more social
psychological vocabulary, they are “attitudes” and “opinions” (things in people’s heads)

Page 8 of 36

PRINTED FROM OXFORD HANDBOOKS ONLINE (www.oxfordhandbooks.com). © Oxford University Press, 2018. All Rights
Reserved. Under the terms of the licence agreement, an individual user may print out a PDF of a single chapter of a title in
Oxford Handbooks Online for personal use (for details see Privacy Policy and Legal Notice).

Subscriber: Goldsmiths College, University of London; date: 02 February 2019

Designing Multimethod Research

and so “words and deeds” are linked to the “attitude behavior” distinction. Attitudes are
seen as “predispositions” that may or may not match “dispositions” or behavior itself.

In comparing words and deeds, the approach is most often to compare prior
(p. 190)

words to subsequent deeds and to ascertain the degree of agreement or divergence

between the two. The classic case of “cognitive dissonance” is but one type of divergence
between word and deed.

Words as data can be gathered in a variety ways that often require direct interaction:
they are “voiced” and may take the form of responses to questions on a survey
instrument, recorded phone conversations, or written documents.

By contrast, deeds as data ideally require direct observation of behavior and so often
require different methods than those used to gather “words.” Hence the multimethod
approach is routinely found in such studies where two different data sets are compared.

Richard Lapiere’s (1934) work is generally cited as being among the earliest multimethod
studies of the relationship between words and deeds or, in his words, attitudes and
actions. In 1930 Lapiere conducted a field experiment consisting of 251 visits across the
United States to assess the responses of hotel and restaurant proprietors to a young
Chinese couple’s requests for service. Only once was the couple turned away. He followed
up this action research component six months later with an attitude component, a mail
survey asking the proprietors whether or not they would “accept members of the Chinese
race” at their establishments. Contrary to their prior actions, very few of the proprietors
expressed an accepting attitude (Lapiere, 1934).

Lapiere’s work has proven to be exemplary in several ways. First, it demonstrated that
individuals’ actions and attitudes could not necessarily be inferred directly from one
another. Second, it demonstrated a methodology for measuring attitudes and actions
separately and then empirically determining their interrelationships. Finally, the study
has inspired and informed later research, particularly in the study of discrimination but
also in the wider theoretical area of attitude–behavior relations (e.g., Ajzen & Fishbein,
1977; Ajzen, 2005). Pager and Quillian (2005) provided an overview of this more recent
and more broadly theoretical work in the introduction to their own research, which is
evocatively titled “Walking the Walk? What Employers Say Versus What They Do.” It is a
sophisticated contemporary analogue of Lapiere’s study. They investigated employment
discrimination based on the applicants’ race and criminal record.

The study was conducted in two stages. In the first “employers’ responses to job
applicants were measured in real employment settings using an experimental audit
methodology. . . . The preferences of employers were measured based on the number of
call-backs to each of the applicants” (Pager & Quillian, 2005, p. 362). “The findings of the
audit showed large and significant effects of both race and criminal record on
employment opportunities” (p. 362). The second stage of the study was a telephone
survey carried out several months later in which employers were asked to express their
hiring preferences for hiring offenders. They found that employers who in the survey

Page 9 of 36

PRINTED FROM OXFORD HANDBOOKS ONLINE (www.oxfordhandbooks.com). © Oxford University Press, 2018. All Rights
Reserved. Under the terms of the licence agreement, an individual user may print out a PDF of a single chapter of a title in
Oxford Handbooks Online for personal use (for details see Privacy Policy and Legal Notice).

Subscriber: Goldsmiths College, University of London; date: 02 February 2019

Designing Multimethod Research

expressed “a greater likelihood of hiring ex-offenders were no more likely to do so in

practice, and that large differences in racial hiring preferences were found in the
experiment, but not in the survey” (p. 362).

Pager and Quillian (2005) concluded that “these comparisons suggest that employer
surveys—even those using an experimental design to control for social desirability bias—
may be insufficient for drawing conclusions about the actual level of hiring against
stigmatized groups” (p. 355).

Rational Action

A central, and some say defining, assumption in classical economic theory is that
individuals act rationally in pursing their own self-interest. From an economic
perspective, words may be seen as a verbal expression of preferences, as illustrated in
surveys of marketing research such as when people are asked which brand of detergent
they prefer. The ultimate behavior or “deed” one is interested in, however, is: “Which
detergent do they buy?” “What do they spend their money on?” According to Paul
Samuelson (1948), who developed the theory of revealed preferences, preferences are
revealed in the behavior of how money is spent. Economists often eschew any other
indicator of preferences other than that. In short, revealed preference is

an economic theory of consumption behavior which asserts that the best way to
measure consumer preferences is to observe their purchasing behavior. Revealed
preference theory works on the assumption that consumers have considered a set
of alternatives before making a purchasing decision. Thus, given that a consumer
chooses one option out of the set, this option must be the preferred option.

(Investopedia, 2014)

One of the critics of the revealed preference theory states that “Instead of
(p. 191)

replacing ‘metaphysical’ terms such as ‘desire’ and ‘purpose’ ” they “used it to legitimize
them by giving them operational definitions” (Wade, 2004, p. 958). Thus in psychology, as
in economics, the initial, quite radical operationalist ideas eventually came to serve as
little more than a “reassurance fetish” for mainstream methodological practice.

A common critique of this rational economic approach is that it is often based on

sophisticated modeling relying on monomethod data points of either an experimental
type, as in “prisoners’ dilemma” research, or monetized aggregated market data. In a
circular self-referential manner, monetary expenditure is the “revealed preference” of
economic researchers about how best to measure rationality itself. When the same
propositions are explored with multimethods, different types of data, and other measures,
researchers are often able to more realistically address the simplifying assumptions of
such models and provide more robust analyses. For example, the seminal work of Amos
Tversky and Daniel Kahneman (1974, 2011) speaks to a variety of different methods that
suggest that economic decision-making may be other than rational. Their research begins
from a psychological experimental approach but is elaborated to include in-depth

Page 10 of 36

PRINTED FROM OXFORD HANDBOOKS ONLINE (www.oxfordhandbooks.com). © Oxford University Press, 2018. All Rights
Reserved. Under the terms of the licence agreement, an individual user may print out a PDF of a single chapter of a title in
Oxford Handbooks Online for personal use (for details see Privacy Policy and Legal Notice).

Subscriber: Goldsmiths College, University of London; date: 02 February 2019

Designing Multimethod Research

interviews, aggregate market data, and historical examples. Their results suggest that
individuals operate in their decision-making by the use of heuristics or rules of thumb
that are often less than optimally rational in economic terms or payoffs.

Mark Granovetter’s (1985) work on the socially embedded nature of economic

transactions again relied on different data and methods that led to a reconsideration of
the “rational individual” assumptions of traditional economics. The study of social
networks required the addition of different methods and types of data beyond that
provided by either aggregated economic or individualistic experimental methods.
Studying the impact of people’s embeddedness in social contexts and social networks has
become both a conceptual and a methodological addition to prior research on
individualized and aggregated data methods and has been used to study everything from
obesity to getting a job or a loan from a bank (Granovetter, 1985; Uzzi, 1999; Cohen-Cole
& Fletcher, 2008).

Contextual Effects

All social behavior occurs in a specific context or situation, and observed patterns of
behavior may be generalized from one context to another, leading to more generalized
understanding, or may be seen to be limited to specific contexts. The idea of seeing
individuals’ behaviors as related to their context is exemplified by W. I. Thomas’s (1923)
older notion of “the definition of the situation” and in more current discussions of
structural effects and in empirical analyses employing hierarchical linear modeling. In
short, individuals’ behaviors may not be simply a product of their individual attributes
and attitudes but may be “influenced” by the fact that they are embedded in socially
specific time and space contexts such as networks of friends and kin, institutions of
school, church, and work, or small and large communities from neighborhoods to nations.
A current example of such a research area is the discussion of “neighborhood effects” on
such specific outcomes as academic achievement, mental and physical health,
delinquency, and employment (Sampson, Morenoff, & Gannon-Rowley, 2002).

The study of such contextual effects often requires an explicit multimethod perspective in
that one is simultaneously gathering data on two different types of units—individuals and
the collective context in which they are embedded. Researchers attempt to assess the
degree to which individual attributes versus collective characteristics (such as “race” or
“class,” for example) have similar or different effects on people’s behavior. A typical
structural effects question would ask: “Do poor people in rich neighborhoods behave
differently than poor people in poor neighborhoods?”

Some subsets or types of contexts whose effects have been explored include the
following:

1. Geo-political units that may vary in scale from neighborhood to cities and metro
areas to states/provinces and nations. (See the voluminous research on
neighborhood effects summarized in Sampson et al., 2002).

Page 11 of 36

PRINTED FROM OXFORD HANDBOOKS ONLINE (www.oxfordhandbooks.com). © Oxford University Press, 2018. All Rights
Reserved. Under the terms of the licence agreement, an individual user may print out a PDF of a single chapter of a title in
Oxford Handbooks Online for personal use (for details see Privacy Policy and Legal Notice).

Subscriber: Goldsmiths College, University of London; date: 02 February 2019

Designing Multimethod Research

2. Organizational/institutional contexts such as schools, firms, agencies (Scott &

Meyer, 1994).
3. Small group/social network/social tie contexts such as friendship, coworkers,
family, and kin.

This is what theorist Georg Simmel (1971) referred to as “social circles” and what Gary
Fine (2012) documented in his numerous ethnographic studies of what he calls “tiny
publics,” ranging from boys’ Little League teams to mushroom hunters to restaurant
kitchens.

The idea of contextual effects is that one is collecting data at different scales, levels, or
units of analysis and these different types of data require (p. 192) different methods of
data collection. For example, at the collective level, there is census or economic data on
geopolitical units or archival data on organizations. These collective levels of data are
then coupled with individual-level data obtained through surveys, interviews, observation,
or records.

Examples of structural effects are seen in the early work of reference group researchers
such as Robert Merton, Samuel Stouffer, and Herbert Hyman among others (Merton &
Lazersfeld, 1950). The classic case is from Stouffer et al.’s (1949), The American Soldier,
where they discovered that satisfaction with the promotion system in different branches
of the military depended on the varying rates of promotion between the branches (Air
Force vs, Marines). The counterintuitive outcome was that higher rates of promotion
produced less satisfaction with the system (Air Force) with lower rates producing greater
satisfaction (Marines). James A. Davis, Spaeth, and Huson (1961) later elaborated these
into systematic comparisons of individual- and collective-level variables producing a
“typology of effects:” (a) Additive effects in which both individual-level and collective-
level variables operate to combine their effects; (b) interaction effects in which the effect
of either the individual-level or the collective-level variable is different depending on the
value of the other; and (c) spurious effects in which the effect of either the individual-
level or the collective-level variable disappears once one has “controlled” for the other
level. More recent techniques of hierarchical linear modeling and their variants similarly
attempt to disentangle these multimethod data to uncover contextual effects
(Raudenbush & Bryk, 2001).

Biomedical Social Research

There is an increasing interdisciplinary perspective on healthcare-related issues that has

led to an explosion of multimethod research using combined methods of collecting
biomedical data, social, and psychological data of individuals. From blood samples to
mouth swabs to more detailed use of medical charts and the numerous “test results” they
may contain, such data can be coupled to interview, survey, or field observational data on
individuals. Such research is often motivated to finding causes or the etiology of disease
or “cures” that may mitigate the consequences. Beyond causal analyses and questions of
social causes of “illness,” or the reverse causation of medical conditions affecting social

Page 12 of 36

PRINTED FROM OXFORD HANDBOOKS ONLINE (www.oxfordhandbooks.com). © Oxford University Press, 2018. All Rights
Reserved. Under the terms of the licence agreement, an individual user may print out a PDF of a single chapter of a title in
Oxford Handbooks Online for personal use (for details see Privacy Policy and Legal Notice).

Subscriber: Goldsmiths College, University of London; date: 02 February 2019

Designing Multimethod Research

and psychological outcomes, lies the measurement and conceptual question of validity of
medical conditions and symptoms. Such assessments may use multimethods research to
actually identify and diagnose or define a syndrome or medical condition, for example
posttraumatic stress disorder.

Such research has become especially significant with increasing use of genetic markers
and some researchers attempts, perhaps too often naively, to link specific genes to
specific behaviors—thereby short-circuiting what are likely much more complex
processes at work. One is reminded of the early attempts of Cesare Lombroso to see a
causal link between skull sizes and shapes and William Sheldon’s “biological body types”
of ectomorphs, mesomorphs, and endomorphs and predilections to engage in criminal/
delinquent behavior.

The multimethod perspective of this research extends beyond different types of data and
data collection to include various analytic frameworks and strategies. Often relying on
very large data sets of tens of thousands of cases, the analytic strategies point to
“statistically significant” correlations between genetic markers and health or behavioral
outcomes, even when the amount of variance explained is miniscule by comparison to the
vast amount of unexplained variance still left to be explained by nongenetic factors. Such
results are too readily overly interpreted from a “genetic determinist” argument by some,
even when the authors of the research caution against such interpretations.

What a multimethod perspective brings to such questions is a need for other types of data
and analyses that would fill in the black box of mechanisms and multifaceted chains of
contingent causation that might link the genetic marker to some specific outcome. The
perennial question of correlation and causation becomes entangled in the nature versus
nurture debate. Jeremy Freese has perhaps been one of the leading sociological
spokespersons calling for more cautionary conclusions based on multimethod strategies,
especially when they are advanced in consideration of policy recommendations and
programs. A recent research article in Science reporting a link between specific genetic
markers and “educational outcomes” cautioned “the main lesson of the research, experts
say, should be that attributing cultural and socioeconomic traits to genes is a dicey
enterprise. “ ‘If there is a policy implication, it’s that there’s even more reason to be
skeptical of genetic determinism’, says Jeremy Freese of Northwestern University in
Evanston Illinois” (Saey, 2013).

How does multimethod research help address genetic/social issues? Research aimed at
relating (p. 193) genetics and social behavior is, of course, inherently multimethod in the
broadest sense of the term because it employs methods derived from biology to study the
genetic components and social science methods for the social components. Moreover,
there are multiple biological research techniques and designs (e.g., blood vs. saliva DNA
tests; twin, sibling, adoption, and molecular genetic studies). However, as sociological
research interest has grown in the past two decades, the area has also become more
multimethod in the general sense of employing varieties of social science methods in
conjunction with one another and also as well with varieties of genetic data.

Page 13 of 36

PRINTED FROM OXFORD HANDBOOKS ONLINE (www.oxfordhandbooks.com). © Oxford University Press, 2018. All Rights
Reserved. Under the terms of the licence agreement, an individual user may print out a PDF of a single chapter of a title in
Oxford Handbooks Online for personal use (for details see Privacy Policy and Legal Notice).

Subscriber: Goldsmiths College, University of London; date: 02 February 2019

Designing Multimethod Research

Survey research is the primary data collecting method currently employed in many social
genetic studies. It allows for large sample studies that may be put to wide use. For
example, the National Longitudinal Study of Adolescent Health was started through the
University of North Carolina–Chapel Hill by Bearman, Udry, and Harris

to capture as much information as possible about the social circumstance,

friendship networks and family conditions of 21,000 teenagers in 132 schools,
from grades 7 through 12. The survey included a disproportionate number of
twins, fraternal and identical, full and half siblings, and adopted children. . .
Follow-up interviews were conducted a year later. Then, for the third wave. . . (in
2002), 2500 siblings were asked for DNA samples (via cheek swabs).

(Shea, 2009, p. B6)

A fourth wave collected DNA with saliva samples for all respondents.

Data from surveys such as this have been employed to investigate the possible main and
interaction effects of genetic characteristics on social behavior and are often
supplemented by additional methods to improve upon the surveys. Guo, Elder, Cai, and
Hamilton (2009), for example, tested the hypothesis that “the genetic contribution to
adolescent drinking depends on the drinking behavior of their friends” (p. 355) with data
from a sample of clusters of siblings and their friends from the larger National
Longitudinal Study of Adolescent Health. They reinterviewed respondents at home,
beyond the survey, to construct the clusters of friends. A special problem in determining
friends’ drinking behavior is that there may be a projection effect when respondents
themselves are asked to report their friends’ behavior. To avoid this possible bias, the
researchers identified and then directly interviewed the respondents’ friends.

In another study, Guo et al. (2009) addressed the issue of selection bias in studying the
influence of friends on behavior in their study of peer impacts on attitudes and drinking
behavior. The study was a survey of a sample of roommates in a large university. The
issue was that adolescents in particular tend to choose as friends people similar to
themselves (Guo et al., 2009). To address this issue, the researchers introduced an
experimental component to their study by separately studying students who had been
randomly assigned to roommates by the university housing office, a procedure that in
their words “avoids the confounding effects of residential choice” (p. 4). The authors
report that “This is the first time that gene-environmental interactions have been
investigated using randomly assigned environmental influences” (p. 6). They conclude
that the “genetic contribution” to adolescent drinking depends on the drinking behavior
of their friends. That is, friend behavior moderates the genetic contribution to alcohol
use.

Page 14 of 36

PRINTED FROM OXFORD HANDBOOKS ONLINE (www.oxfordhandbooks.com). © Oxford University Press, 2018. All Rights
Reserved. Under the terms of the licence agreement, an individual user may print out a PDF of a single chapter of a title in
Oxford Handbooks Online for personal use (for details see Privacy Policy and Legal Notice).

Subscriber: Goldsmiths College, University of London; date: 02 February 2019

Designing Multimethod Research

Evaluation Research

Multimethod research has found a ready application in the burgeoning area of evaluation
research. Evaluation research emerged explicitly from a policy-oriented set of questions
about the effectiveness of programs designed to deal with specific social problems (Rossi,
Lipsey, & Freeman, 2004). The call for evaluation of such programs has been heightened
under the current antitax political context of doing away with programs that cannot
demonstrate their effectiveness and more stringently their efficiency in a cost/benefit
calculus. Evaluation research is therefore explicitly geared to providing data and analysis
that can inform decision-making about supporting or negating specific programs. As a
rubric in rational decision-making quantification is seen to be particularly valued as able
to provide objective, universalistic, summary, comparative criteria by which to judge such
programs. Coupled with the “natural” quantification provided by a “money” calculus,
costs and benefits can be rendered commensurable and evaluated accordingly (Espeland
& Stevens, 1998).

An additional more refined consideration beyond assessing efficacy and efficiency is that
of equity. Here the question becomes: “Who bears the cost, and who reaps the benefit?”
The answer to this question may require different kinds of data and still different
methods of research to distinguish how the costs and benefits are differentially
distributed.

This logic of evaluation research has had a number of consequences. One is to

(p. 194)

focus on outcomes, more explicitly outcomes that are quantifiable, and even more
explicitly quantifiable outcomes that can be given a dollar value. Second, the most valued
research design is the true experiment with pre- and posttest measurement of outcomes
(or “dependent” variables) coupled with random assignment to control and experimental
groups that “do” or “do not” get the treatment of the program (the independent variable).
These “ideal” research designs are often difficult to implement for any number of cost,
time, practical, and ethical reasons, and this has led to a variety of quasi-experimental
designs that relax one or another of the criteria (Campbell & Stanley, 1963, Cook &
Campbell, 1979; Cook, 2002; Shadish, Cook, & Campbell, 2002). The actual
implementation and process of carrying out the program are often left relatively
unexplored in such designs. This is the point at which more qualitative and ethnographic
methods are brought to bear in a fuller understanding of the evaluation of programs
(Cook, Shadish, & Wong, 2008). Such qualitative research (e.g., in-depth interviews and
field observation) may provide an understanding of the “causal process,” the detailed
concatenated sequential “mechanisms” by which certain outcomes were obtained.
Furthermore, such qualitative research may heighten the possibility of serendipitous
discovery of unanticipated outcomes not previously considered, especially given a specific
focus on monetized quantitative measurable outcomes.

As a result of the these considerations, evaluation research has increasingly come to

incorporate a multimethod perspective—combining a variety of methods that more
robustly evaluate programs in terms of process and outcomes and varieties of methods

Page 15 of 36

PRINTED FROM OXFORD HANDBOOKS ONLINE (www.oxfordhandbooks.com). © Oxford University Press, 2018. All Rights
Reserved. Under the terms of the licence agreement, an individual user may print out a PDF of a single chapter of a title in
Oxford Handbooks Online for personal use (for details see Privacy Policy and Legal Notice).

Subscriber: Goldsmiths College, University of London; date: 02 February 2019

Designing Multimethod Research

both within and between qualitative and quantitative methods. The “approximations” to
what some see as the ideal of randomized experimental designs in “quasi-experiments
come with appropriate caveats as to their limitations (Cook & Campbell, 1979), and as
Cook et al. (2008) later note, the use of different methods in combination have utility for
answering a variety of different questions in evaluation research.

A number of major experimental social policy programs have been evaluated using
multimethod research. For example researchers at the Seattle Income Maintenance
Experiment and the Denver Income Maintenance Experiment randomly assigned poor
people to various regimes and levels of welfare expenditure and assessed through
multiple methods, such as surveys, interviews, diaries, and various specific outcomes in
the life and welfare of recipients (US Department of Health and Human Services, 1983).
More recently, Move to Opportunity Programs have been evaluated with a variety of
methods to assess the effects of moving poor families out of concentrated inner-city
public housing to better neighborhoods and assessing various outcomes such as health,
employment and educational performance of children (Briggs, Popkin, & Goering, 2010).

Greene, Caracelli, and Graham (1989) developed a systematic rationale and prescription
for the use of multimethods in evaluation research for five distinct purposes: (a)
triangulation for convergence and corroboration, (b) complementarity for elaboration and
enhancement, (c) development for helping to inform one method by another, (d) initiation
for discovering paradox and contradiction, and (e) expansion for extending breadth and
range. These were assessed empirically by looking at 53 evaluation studies and resulted
in recommendations for ideal mixing of particular combinations of qualitative and
quantitative methods. The combinations took into account their similarity and difference
in phenomena studied, paradigm used, primacy of one over the other, integration or
independence, and timing. One of the key distinctions noted was between product
(outcome)-oriented evaluation and process-oriented evaluation and the finding that those
that evaluated both product and process constituted the majority of mixed method
studies.

Comparative Historical Causation, Trends, and Character Types

Our initial formulation of the multimethod perspective in multimethod research (Brewer

& Hunter, 1989) clearly called for a broad conception of multimethods beyond the narrow
question of measurement as in Campbell and Fiske’s (1959) landmark article on
triangulation. Multimethods we suggested should apply to all stages of the research to
include for, example, multiple theoretical perspectives, and also different analytical
strategies.

Historical causation
The question of varying analytical strategies has emerged in a number of arenas of social
science but perhaps no more clearly than in the field (p. 195) of comparative/historical
analysis. This becomes especially apparent in addressing the question of causation and its
complexities (Abbott, 1990). Should one pursue a variable-based statistical linear

Page 16 of 36

PRINTED FROM OXFORD HANDBOOKS ONLINE (www.oxfordhandbooks.com). © Oxford University Press, 2018. All Rights
Reserved. Under the terms of the licence agreement, an individual user may print out a PDF of a single chapter of a title in
Oxford Handbooks Online for personal use (for details see Privacy Policy and Legal Notice).

Subscriber: Goldsmiths College, University of London; date: 02 February 2019

Designing Multimethod Research

approach of a large numbers of cases isolating the impact of causal variables on selected
outcomes, or should one pursue a small-N comparative analysis based on detailed
exposition of selected cases using a logic of similarity/difference and presence/absence
and being sensitive to sequence and timing in trying to isolate necessary and sufficient
causes (Skocpol & Somers, 1980; Skocpol, 1984)? This is where Andrew Abbott’s (1997)
reflections on the early Chicago School stressing “contingency and context” suggest the
need for multimethod strategies. Abbott distinguished, for example, between those
comparative historians who rely on variable analyses of large-N studies of nation/states
statistically parsing out the effects of independent variables on selected dependent
variables of interest versus those who rely on a small-N more comparative case-based
analysis of specific events that have unfolded at varying points in time.

Among comparative historical analysts, Charles Ragin (1987) has developed a synthetic
approach that reflects a multimethod merger of qualitative and quantitative methods. He
uses the logic of Boolean algebra and “fuzzy sets” that produces a rigorous quantifiable
assessment of varying combinations of qualitative characteristics that permit one to
assess the contribution of necessary and sufficient causes for given outcomes. In short,
he marries the rich detailed knowledge of qualitative data and small-N comparative logic
with larger N data sets and develops quantitative statistical assessments of the likelihood
of different combinations of contexts and conditions producing specified outcomes.
Walton and Ragin (1990) have applied this technique to a study of which combinations of
factors contributed to countries experiencing middle-class riots in response to
International Monetary Fund policies. James Mahoney and Dietrich Rueschemeyer (2003)
posed similar concerns in their landmark compilation Comparative Historical Analysis in
the Social Sciences. Focused especially on questions of causation such as necessary and
sufficient causes and temporal order, sequence, and duration in societal level events—
such as revolutions and depressions—they were concerned with the different logics and
modes of analysis in this emergent field. Specifically addressing the concerns that a
multimethod strategy is designed to address, they noted

First of all, across the relevant disciplines, methodological disagreements of

varying intensities have emerged between qualitative and quantitative
approaches. In comparative social science this is represented by disputes between
comparative historical researchers and cross-national statistical researchers who
work with large numbers of cases. (p. 16)

However, they note that the advocates of large-N and small-N methods are more recently
“acknowledging that there is a place for both in the cycle of research” (p. 17). For
example, methodologists report on iterative research programs in which comparative
historical research supplements the initial findings of statistical studies and vice versa.
They quote King, Kehone, and Verba’s book Designing Social Inquiry (1994) to the effect
that “the differences between the quantitative and qualitative traditions are only stylistic
and are methodologically and substantively unimportant. All good research can be
understood—and is indeed best understood—to derive from the same underlying logic of
inference” (p. 4). Mahoney and Rueschemeyer (2003) conclude with an invocation to the

Page 17 of 36

PRINTED FROM OXFORD HANDBOOKS ONLINE (www.oxfordhandbooks.com). © Oxford University Press, 2018. All Rights
Reserved. Under the terms of the licence agreement, an individual user may print out a PDF of a single chapter of a title in
Oxford Handbooks Online for personal use (for details see Privacy Policy and Legal Notice).

Subscriber: Goldsmiths College, University of London; date: 02 February 2019

Designing Multimethod Research

spirit of multimethod research “comparative historical researchers do their best research

when they remain open to the use of diverse methodologies and analytic tools” (p. 24).

Trends
Debates over causal analysis in comparative and historical social science are often
predicated on documenting broad historical trends of social change such as urbanization,
industrialization, bureaucratization, and secularization, all of which are often captured in
theories of modernity (Moore, 1958, 1966). Such studies—both historical analyses of
changes over time and comparative analyses of different societies considered to be at
different stages in an historical linear development from primitive to modern—often rely
on a variety of data and different methods to demonstrate these broad trends of social
change. We are not debating here the “Western bias” of theories of modernity or their
assumptions of linear societal development (see Wallerstein, 1974, 2004 on world systems
theory and others for such questions and critiques), nor are we entering the debate as to
whether or not such trends can be variously couched as upward positive “progress”
versus downward declining trajectories, or combinations in various cyclical rise and fall
theories. Rather we are highlighting here (p. 196) the degree to which such studies often
rely on a multimethod accumulation of diverse sets of data as the basis of their
assertions. We will briefly look at two such empirically based landmark exemplars, David
McClelland’s (1961) research reported in The Achieving Society and Robert Putnam’s
(2001) more recent research on the decline of civil society reported in Bowling Alone.

As a social psychologist, McClelland (1961) became fascinated with the psychological

trait that he identified as “the need for achievement” or nAch as it became known, which
is an internalized motive of people to continue to strive for ever greater accomplishments.
Drawing on the work of the sociologist Max Weber (1905) and his landmark study of the
rise of modern capitalism documented in The Protestant Ethic and the Spirit of
Capitalism, McClelland saw this need for achievement as a cultural variable that
socialized individuals from different cultures into varying needs for achievement. In his
experimental research, he documented the ways in which this need for achievement
could be manipulated and heightened or depressed under varying conditions. But he
extended his line of research far beyond the laboratory to include other data and other
methods, such as analyses of artifacts of different cultures over time and content analysis
of myths and stories from different cultures as to their emphasis on “achievement.” He
correlated these with data from still other methods such as governmental statistics and
showed, for example, a correlation between socialization into need for achievement as
indicated by children’s stories and GDP as an outcome variable reinforcing one of
Weber’s central tenants of the Spirit of Capitalism.

Robert Putnam (2001) begins his analysis of the decline of civil society in America by
focusing on the questions of “trust” and “trustworthy,” which are seen to rest ultimately
on the idea of keeping one’s word—meaning one’s subsequent behavior is in line with
one’s prior verbal commitment. Trust is not just an attribute of individual dyadic
relationships but may be generalized not only to people in general, including even
strangers, but to institutions as well, as in people’s trust in government. The idea of civil
Page 18 of 36

PRINTED FROM OXFORD HANDBOOKS ONLINE (www.oxfordhandbooks.com). © Oxford University Press, 2018. All Rights
Reserved. Under the terms of the licence agreement, an individual user may print out a PDF of a single chapter of a title in
Oxford Handbooks Online for personal use (for details see Privacy Policy and Legal Notice).

Subscriber: Goldsmiths College, University of London; date: 02 February 2019

Designing Multimethod Research

society, that is, voluntary social relationships among individuals who have minimal social
ties and may even be strangers to one another, is predicated on this element of a
presumption of trust. To explore this realm of informal voluntary social ties, that is, civil
society, which is distinct from primary ties of friends and kin and more formal ties
governed by laws and the institutions of social control of the state (Shils, 1957), requires
a variety of different methods and data sources. This is clearly demonstrated by Putnam
in his contemporary classic, Bowling Alone: The Collapse and Revival of American
Community. In this summary volume, he argues there has been a decline and
transformation of the civil sphere in American life. In building his argument he presents a
variety of data derived from different methods, including survey research on participation
in voluntary associations, attitudes toward government, and generalized trust of others,
as well as historical and archival data such as organizational memberships, voting
records, and content analyses of media. As he states in his methodological appendix in
Bowling Alone, “My primary strategy, as explained in Chapter 1, has been to triangulate
among as many independent sources of evidence as possible, following the model of
researchers into global warming” (p. 315). Overall he finds there is a decline in people’s
expressed trust of major traditional institutions in society –from government, to religion
to business—which is seen to be the result of broken promises in the conduct of these
institutions due to incompetence and corruption that undermines their legitimacy. It is
the sheer volume and variety of data derived from different methods that makes his
argument and analysis so persuasive. This is in part because he is able to provide a litany
of data that support his argument and also because of the variety and types of data
through which he is able to demonstrate a subtle and nuanced analysis that shows
exceptions, qualifications, and limitations that increase the credibility of his overall
argument. In short, he comes across as an objective researcher rather than a true
believer polemicist.

Character types
C. Wright Mills (1959) in The Sociological Imagination roots the understanding of social
life in the intersection of biography and history. Combining these two elements has been
a continuing concern in social science research and has often involved different data and
methods—a multimethod perspective—to capture and tease out this critical intersection.
We have already briefly looked at macro-level multimethod approaches to history in
considering causation and trends. Cultural shifts over time have social consequences and
vice versa, not the least in producing different emerging types of characters as a
consequence.

With respect to biography, there is an old tradition within sociology of producing life
histories or (p. 197) characterizations of a type of person: a composite summarization of
data obtained from numerous discrete individuals. These include The Unadjusted Girl
(Thomas, 1923), The Jack-roller (Shaw, 1930/2013), The Professional Thief (Sutherland,
1937), The Hobo (Anderson, 1923), and others. James Bennett (1981) wrote of this
approach as a typification or characterization of what theorists like Robert Merton (1968)
would call statuses and roles, that is, structural positions that are distinct from the more

Page 19 of 36

PRINTED FROM OXFORD HANDBOOKS ONLINE (www.oxfordhandbooks.com). © Oxford University Press, 2018. All Rights
Reserved. Under the terms of the licence agreement, an individual user may print out a PDF of a single chapter of a title in
Oxford Handbooks Online for personal use (for details see Privacy Policy and Legal Notice).

Subscriber: Goldsmiths College, University of London; date: 02 February 2019

Designing Multimethod Research

individualistic psychology of personality. The composite character was a fictionalized

distillation and amalgamation, with data selectively organized and summarized, but it was
all true in the sense that all of the facts were verified and valid or, as one commentator
put it, “nothing in here was made-up” (Howard Becker in personal communication at
Rhetoric of Research Conference Northwestern University, May 1980). This is one type of
biography that may result from an analysis of numerous discrete individual biographies.
By contrast, the focus on discrete individual biographies—the archetypal literary
biography such as James Boswell’s The Life of Samuel Johnson (1998) or William
Manchester’s (1978) American Caesar: Douglas MacArthur, 1880–1964—are in effect
case studies of a single individual while the composite biography constitutes the case
itself (see Ragin and Becker’s What Is a Case? [1992]). Here, different methods and
different types of data would be employed to produce the distinction between an
individual personality and a social type or character.

The former would more likely involve archives and perhaps interviews of those who knew
the focal person, while the latter would more likely draw on interviews and observations
of numerous individuals representative of that type. An example of the latter is Martin
Jankowski’s (1991) development of the summary gang character defined by defiant
individualism (as opposed to a specific individual’s personality). He synthesized and
distilled the characteristics of a typical gang member based on his decade-long
participant observation of 29 gangs in three cities. A focus on character requires a
multimethod strategy that incorporates cultural (e.g., ideas, beliefs) and structural (e.g.,
network) data about the milieu within which character types are formed. Context
produces similarity that is distilled in similar character traits. Fitting individuals into
history is a multimethod strategy of placing different types of characters into different
historical trends and causal processes. The life histories are typified characters who are
the outcomes of these processes.

Disaster Research

Disaster research is one of the earliest sociological research areas to develop an explicitly
multimethod approach. The impetus was the immediacy and transience of the events
being studied.

On December 5, 1917, a munitions ship exploded in Halifax Harbor, inflicting severe

damage on the city and injury to its inhabitants. Close to 2,000 people were killed and
9,000 more injured (in all about 22% of the city’s population). Samuel Henry Prince, who
was then serving as an episcopal priest in Halifax, was both a participant and an observer
of this explosion and its results. He later recorded his observations in his Columbia
University doctoral dissertation, in what is generally regarded as the first sociological
study of disaster (Prince, 1920). Prince introduced his study with a methodological note
that has resounded in the design of disaster research ever since:

The whole field. . . is a virgin subject in sociology. Knowledge will grow scientific
only after the most faithful examination of many catastrophes. But it must be

Page 20 of 36

PRINTED FROM OXFORD HANDBOOKS ONLINE (www.oxfordhandbooks.com). © Oxford University Press, 2018. All Rights
Reserved. Under the terms of the licence agreement, an individual user may print out a PDF of a single chapter of a title in
Oxford Handbooks Online for personal use (for details see Privacy Policy and Legal Notice).

Subscriber: Goldsmiths College, University of London; date: 02 February 2019

Designing Multimethod Research

realized that the data of greatest value is left sometimes unrecorded and fades
rapidly from the special memory. Investigation is needed immediately after the
event. (p. 22)

In short, disaster studies to capture these fleeting memories and events must be based on
quick response and firsthand research.

Following the disastrous experiences of World War II and the onset of the Cold War (Form
& Nosow, 1958), a number of quick-response disaster research programs were
established at the National Opinion Research Center and elsewhere (see Barton, 1969). In
1963 the Disaster Research Center was established to more regularly monitor disasters
with field and survey research. It has conducted over 600 such studies since its inception.
Thus, in contrast to the 1917 Halifax explosion, the September 11, 2001, attack on New
York City’s World Trade Center, while equally unexpected, had trained and experienced
observers from the Disaster Research Center ready to respond almost immediately.

However, the routinization of disaster research readiness does not routinize the
disastrous events themselves or the research into them. For example, one of the lead
Disaster Research Center 9/11 researchers writes in her dissertation:

Direct observation of ongoing emergency activities was particularly valuable in

this case precisely (p. 198) because the event itself seriously hampered record-
keeping and clouded the memory of some emergency responders. With key
decision makers often unable to recollect what was happening literally from one
moment to the next, the ability to actually be present as decisions were made was
critical for later efforts to reconstruct events. . . also facilitated rapport in later
face-to-face interviews and added validity to information obtained in those
conversations. To contend with the. . . research challenges. . . I draw upon the
variety of data sources and employ multiple qualitative strategies—among them
direct exploratory observation, primary data collection and analysis, as well as in-
depth face-to-face and telephone interviews—to triangulate emergency response
information.

(Kendra & Wachtendorf, 2003, pp. 41–42)

More generally, Stallings (1997) has suggested that

The “challenge” of disaster research. . . is the lack of time between the occurrence
of a disaster and the fielding of research: lack of time to develop theory and
hypotheses; lack of time to develop research instruments; lack of time to decide
which events are worthy of study. (p. 9)

This highly specialized area of research highlights another key factor that a multimethod
perspective has proven be helpful in addressing the time element and a need for
pragmatic quick response to an often unanticipated research situation and the need to
address the multiple audiences and consumers of research.

Page 21 of 36

PRINTED FROM OXFORD HANDBOOKS ONLINE (www.oxfordhandbooks.com). © Oxford University Press, 2018. All Rights
Reserved. Under the terms of the licence agreement, an individual user may print out a PDF of a single chapter of a title in
Oxford Handbooks Online for personal use (for details see Privacy Policy and Legal Notice).

Subscriber: Goldsmiths College, University of London; date: 02 February 2019

Designing Multimethod Research

One example of this pragmatic adaptation in a manmade disaster was the “experimental/
survey” research of Bobo, Zubrinsky, Johnson, and Oliver (1994) in Los Angeles during
the time of the Rodney King riot following the “not guilty” verdict of police officers
captured on tape beating Mr. King. A survey had been launched days prior to the riot,
which, among other things, had a series of items focused on race relations. After the riots
the survey continued, and the researchers had a “natural experiment” or quasi-
experimental design of before and after measures with the riot as the experimental
variable. This piece of research beautifully represents two key aspects of the spirit of
multimethod research—a pragmatic adaptive response to the serendipity of the research
situation and a melded multimethod design combining survey and experimental logics.

The fact that “disaster research” often has multiple purposes for different audiences is
another factor that makes multimethod research particularly attractive as a purposeful
design. This was clearly the case, for example, in Kai Erikson’s (1976) research following
the Buffalo Creek disaster in West Virginia reported in Everything in Its Path: Destruction
of Community in the Buffalo Creek Flood. Erikson was initially approached by a law firm
involved in the litigation of the case on behalf of residents whose homes and hamlets had
been wiped out by the floodwaters released by a failed earthen dam built by a mining
company. Erikson’s multimethod research relied predominantly on in-depth interviews
with displaced residents and, perhaps most poignantly and empathetically, firsthand field
observations of the everyday disrupted lives of residents of the hills and hollows of the
area. Documenting the effects of the disaster for the legal audience may have been the
initial impetus of the research, but other audiences included welfare services and various
governmental agencies. Drawing on officially compiled statistics and archives, including
the history of actions by the mining company, permitted Erikson’s multimethod research
to both draw from and be of use to multiple sources and audiences. The simple profound
conclusion for the sociology of disasters was to qualify the commonly stated proposition
that disasters create a heightened sense of community in response to the shared fate.
Erikson’s research concluded that if the disaster was sufficiently severe, as in the case of
Buffalo Creek, it could in fact destroy community.

Another example of disaster research employing multimethods is Eric Klinenberg’s (2003)

study of the Chicago summer heat wave of 1995, which resulted in the death of hundreds
of Chicagoans, especially the elderly in certain poor minority neighborhoods. Again,
relying on official statistics, in-depth interviews, and field observations Klinenberg
concluded that the deaths were differentially distributed by neighborhood and that a
critical factor was the variable density of networks of elderly found in two adjacent
neighborhoods—dense in a Latino community that had relatively fewer deaths and
relatively sparse in an adjacent Black community that had greater deaths. The relative
isolation or connectedness affected whether or not others were there to check on how the
elderly were faring in the heat and to offer assistance if needed. In a broadened (p. 199)
autopsy of the disaster, Klinenberg’s research also explored the response and programs
in place of various city agencies and departments, from police and fire to health and
welfare, and concluded that they too bore some culpability.

Page 22 of 36

PRINTED FROM OXFORD HANDBOOKS ONLINE (www.oxfordhandbooks.com). © Oxford University Press, 2018. All Rights
Reserved. Under the terms of the licence agreement, an individual user may print out a PDF of a single chapter of a title in
Oxford Handbooks Online for personal use (for details see Privacy Policy and Legal Notice).

Subscriber: Goldsmiths College, University of London; date: 02 February 2019

Designing Multimethod Research

One policy implication of the research has now been instituted by the mayor’s office:
repeated public service announcements during subsequent heat waves for residents of
the city to contact and check on the welfare of elderly kin and neighbors and for city
employees such as police and fire and welfare workers to do the same. Klinenberg’s
(2003) research has also been important for thinking about the quality of multimethod
research and raising the question: Does the use of multimethods ipso facto produce
better research results? Critiques of Klinenberg’s research by Duneier (2004, 2006) and
others generated some controversy and debate as to the adequacy and “thinness” of the
field research he conducted. The implication is that time and effort spent on collecting
other types of data might have been better spent in a greater immersion in the field
providing more “thick description” rather than the more superficial observations
reported. The lesson is, in short, that multimethods may be valuable but that simply
having more methods and different data does not ensure greater validity if the methods
themselves are inadequately employed.

Disaster research often attempts to distinguish between what is natural, as in a natural

disaster, and what is the result of human agency and in the latter case to parse out
“blame” and accountability This research gets readily translated into legal conceptual
frameworks of culpability. Recent legal cases, for example, have held meteorologists
“guilty” for having failed to predict and warn residents of impending storms or geologists
for failure to warn of earthquakes (Erikson, 1994; Freudenburg, Gramling, Laska, &
Erikson, 2009).

The art of translating scientific and social science research results to multiple audiences
was exemplified in the case of research on Love Canal, the toxic chemical site in Upstate
New York where different publics, which included fellow scientists, evacuated former
residents eager to learn if they could safely return to their homes and public officials and
governmental decision-makers all eager for a simplified “yes or no” recommendation.
Different data obtained through different methods, and perhaps equally significant, from
different disciplines proved useful in communicating an understanding of the facts of the
case and their implications for what people might decide to do (Hunter, 1986).

Rhetoric, Narrative, and Postpositive

Postmodern Multimethods: Turn, Turn, Turn
Science is a social process, and social science is social in both form and substance. The
idea that methods of systematic research are employed to discover truth may be a central
and noble teleological goal, but it is an oversimplified, reified, and idealized conception of
what we actually do. We are not mere scientific automatons programmed to follow fixed
procedures for probing reality, like the Mars rovers (Spirit, Opportunity, and Curiosity),
but rather we are active social agents who talk to one another, read one another’s work,
and debate and argue about the direction, meaning, and credibility of one another’s
research and our assertions about the “truth” of what we have observed. This social
constructionist perspective on science has a number of implications often defined as
Page 23 of 36

PRINTED FROM OXFORD HANDBOOKS ONLINE (www.oxfordhandbooks.com). © Oxford University Press, 2018. All Rights
Reserved. Under the terms of the licence agreement, an individual user may print out a PDF of a single chapter of a title in
Oxford Handbooks Online for personal use (for details see Privacy Policy and Legal Notice).

Subscriber: Goldsmiths College, University of London; date: 02 February 2019

Designing Multimethod Research

postmodern or postpositivist critique. Two of the more significant ones are the role of
“the new rhetoric” and the “narrative” emphasis that sees science as a collection of
stories. We briefly explore elements of rhetoric and narrative in turn.

The Rhetorical Turn

The social and rhetorical aspect of science is evidenced in a multimethod perspective

with respect to the following:

1. The social goal of science is to convince others of the tentative truth of one’s
assertions about questions posed. The tentativeness arises for a number of reasons,
such as method limitations and historical and spatial limitations. These result in
varying degrees of generalizability—from modest, limited claims of local truths to
broader claims and heroic ones of universal truths. The convincing part entails the
art of persuasion and being attentive to one’s audiences and the criteria by which
they will judge the validity of one’s assertions.
2. Multimethods permit a variety of different questions to be posed about a given
phenomenon of interest. For example, is the question/assertion one of descriptive
fact or causal explanation (correlation vs. causation), one dealing with macro
structural/cultural phenomena or the micro level of agency and intent, one focused
(p. 200) on process and outcome, or one concerned with understanding alone or

prescriptive policy implications?

3. Diverse others may have a variety of criteria by which they evaluate and are
convinced of the truth value of one’s assertions. Multimethods allow one to address
these different criteria. Different methods are varyingly adept at addressing each of
the skeptic’s questions, and beyond these there may be other criteria in addition—
some more general and some more specific that go to ontological assumptions: for
example debates over the best mode of analysis in studying social change from
“interrupted time series with switching replications” (Cook & Campbell, 1979) to
event history analysis (Allison, 1984) to patterned sequences of nomothetic narrative
(Abbott 1995).
4. Multimethods address diverse criteria and answer a variety of skeptic’s questions,
thereby becoming more convincing to more people and more types of people, as
different types of texts and rhetorics are employed. These others may range from
fellow academics concerned with the skeptics’ questions to policy planners
concerned with cost/benefit analyses of effectiveness and efficiency to the broader
public concerned with issues of equity and ethics. The variety of rhetorical
arguments contained in a multimethod perspective may range from quantitative
survey results to numbers to a sample of personal accounts to historical archives.

The rediscovery of rhetoric as a central component of scientific argument and social

research is based on this social aspect of science. From these rhetorical perspective
methods and the varying rules, procedures, techniques, and norms of science we derive
tools for building an argument about the link between observations and data about the
world and one’s ideas or theories. Scientific method narrowly construed is concerned

Page 24 of 36

PRINTED FROM OXFORD HANDBOOKS ONLINE (www.oxfordhandbooks.com). © Oxford University Press, 2018. All Rights
Reserved. Under the terms of the licence agreement, an individual user may print out a PDF of a single chapter of a title in
Oxford Handbooks Online for personal use (for details see Privacy Policy and Legal Notice).

Subscriber: Goldsmiths College, University of London; date: 02 February 2019

Designing Multimethod Research

about “persuading skeptics” by satisfying them that one has followed these rules in the
research. According to The British Strong Programme (Bloor, 1976/1991; Mulkay 1979).
science is, from this perspective, subsumed under rhetoric as a method or “tool” of
argumentation and persuasion. The “strong” assertion is that it is all rhetoric through and
through. Hunter (1990) in The Rhetoric of Social Research sees social science research as
a set of rhetorical relationships among the researcher, the subjects, and the audience tied
together by texts or research reports.

Not only are there “conventions” of science that must be followed to be convincing, but
the reports of research, from published articles to PowerPoint presentations, must be
similarly stylized to persuade specific audiences. The multimethod approach, which is a
“synthesis of styles” of research (Brewer and Hunter, 1989), lends itself to being more
persuasive (believable) not merely due to a greater number of methods but to their
addressing different criteria of credibility. As poet John Donne noted about An Obscure
Writer

Philo, with twelve years study hath been grieved

This is to be understood as when will he be believed.

(Donne, 1986, p. 210)

Margarete Sandelowski (2003) in Tables or Tableaux? The Challenges of Writing and

Reading Mixed Methods Studies distinguished between method and methodology and
also paradigms and techniques and added, “A major—and arguably the most important—
criterion in evaluating the merits of a study lies in the ability of writers to persuade
readers of its merits in their research reports (p. 321). Invoking Fish (1980), she directly
addressed the rhetoric of multimethods by noting that qualitative and quantitative belong
to different interpretive communities. She considered this an “aesthetic criteria,
including the sense of rightness and comfort readers experience that is crucial to the
judgment they make about the validity of a study.” She therefore explores the need for”
mixed media for mixed methods for mixed audiences” (p. 335).

The issue of different genres of research reports are seen in two chapters from Hunter’s
edited volume (1990)—Joseph Gusfield’s (1990) comparison of Liebow’s (1967) Tally’s
Corner based on qualitative research versus Blau and Duncan’s (1978) quantitative
analysis of the American Occupational Structure—and in Marjorie DeVault’s (1990)
comparison of Kanter’s (1993) Men and Women of the Corporation versus Krieger’s
(1983) The Mirror Dance, both dealing with women’s roles and identities bur the former
with a traditional organizational analysis and the latter with a more feminist perspective.
To paraphrase DeVault: same subject, different methods. Quoting Wolfer (1991),
Sandelowski (2003) noted that “different aspects of reality lend themselves to different
methods of inquiry” (p. 327), and, she added, “there is no uniform paradigm-method link,
there is a method reality link” (p. 327).

Page 25 of 36

PRINTED FROM OXFORD HANDBOOKS ONLINE (www.oxfordhandbooks.com). © Oxford University Press, 2018. All Rights
Reserved. Under the terms of the licence agreement, an individual user may print out a PDF of a single chapter of a title in
Oxford Handbooks Online for personal use (for details see Privacy Policy and Legal Notice).

Subscriber: Goldsmiths College, University of London; date: 02 February 2019

Designing Multimethod Research

Sandelowski (2003) further suggests the mixing may have one of two purposes “to
(p. 201)

achieve a fuller understanding of a target phenomenon and to verify one set of findings
against the other or a comprehensive kaleidoscopic sense of understanding versus truth
or validity ‘representation’.” She posed a great question: What is mixed? And what kind of
mixing occurs? Through focusing on the qualitative quantitative distinction she
recognizes the ambiguity in distinguishing between them and the difficulty of crafting
research reports incorporating both in a convincing manner.

Johnson and Turner’s (2003) chapter in the Handbook of Mixed Methods uses the key idea
of “trustworthy,” which they equate with validity and which in the title of Hunter’s book
would be “believable.” They say, “Valid research is plausible, credible, trustworthy, and,
therefore, defensible. . . . We treat the terms valid and trustworthy as synonyms” (p. 300;
italics in original). This may be directly linked to multimethods in that we are prone to
trust the many over the few, if confirmatory, and to have if not “mistrust” then at least
doubt over the disconfirming or specification of differences due to method itself (Lever,
1978).

Narrative

The rhetorical turn is closely allied with another postmodern humanistic approach to
social science research and that is the idea that what we do is basically tell a story—
create a narrative. The iconic corollary to multimethod research from the narrative
perspective is like the “Rashomon effect,” the stories of a single event or phenomenon
told from the perspectives of different witnesses and participants. This, of course, raises
the larger phenomenological question—is it many versions of one event or many different
events? We operate from the assumption that the real world does exist (physicalism) and
yet recognize that our knowledge of that real world is a varied and imperfect product of
the observations we make of it (relativism).

As Andrew Abbott (1992) has observed in his landmark paper “From Causes to Events:
Notes on Narrative Positivism”:

In the last decade, a number of writers have proposed narrative as the foundation
for sociological methodology. By this they do not mean narrative in terms of words
as opposed to numbers and complexity as opposed to formalization. Rather, they
mean narrative in the more generic sense of process, or story. . . . In the context of
contemporary empirical practice, such a conception is revolutionary. Our normal
methods parse social reality into fixed entities with variable qualities. They
attribute causality to the variables –hypostatized social characteristics—rather
than to agents; variables do things, not social actors. Stories disappear. (p. 428)

The key trope (Booth, 1961/1983) that governs narratives of science, including social
science, is that of a “quest.” It is a quest that begins with a question leading to a search
using research to explore the unknown. It is a purposeful journey to find a treasured goal,
to go out into that real world and make the unknown known—in a word to acquire
knowledge. One hopes that the knowledge one acquires has some validity that it is “true”
Page 26 of 36

PRINTED FROM OXFORD HANDBOOKS ONLINE (www.oxfordhandbooks.com). © Oxford University Press, 2018. All Rights
Reserved. Under the terms of the licence agreement, an individual user may print out a PDF of a single chapter of a title in
Oxford Handbooks Online for personal use (for details see Privacy Policy and Legal Notice).

Subscriber: Goldsmiths College, University of London; date: 02 February 2019

Designing Multimethod Research

and that what one believes to be true about the phenomena bears some close
approximation to reality. From a pragmatic perspective—it is valid if the knowledge
works, if it allows one to accomplish what one wishes to get done; if it doesn’t work it is
useless (Dewey, 1938).

Some might think the narrative is merely the story about how one does science—that it is
not the doing of science itself. We suggest that the narrative is in fact a critical part of the
doing of science itself, and the stories we tell about what we do serve to reflexively
construct and reconstruct the actual quest itself.

As noted by Scott Baker (1990):

In studying the discourse of scientific communities, in fact, we find these

communities employing multiple rhetorics as often as they use strict singular
logics. . . . We note moreover, that scientists do not exclusively depend upon, or
even follow, the strict guidelines of their field’s logic or method. They convince
each other and the lay public that their theories are reliable by means of
persuasion not provided for or sanctioned by the accepted methodology. . . these
communication strategies are outside the formal rules of method, [yet] they do
persuade. (pp. 233–234)

But the postmodernist idea of narrative is more than just telling a story about how
science is conducted; narrative is seen to be an ontological aspect of how we as “sensate
human beings” “make sense” of the real world. Abbott (1997) claims we do so by taking
into consideration “context and contingency,” and Ragin (1987) suggests we tell
narratives of constellations of characteristics defining our objects of study (our cases) as
they alter through (p. 202) time. To highlight a quote presented previously that focused on
the mechanism or the black box of explanation (Guo & Adkins, 2008):

It should be pointed out that significant statistical findings alone are rarely, if ever,
considered proof of a link between a genetic variant and a human complex
phenotype. This contrasts with the usual practice in social sciences. Repeated
significant results in social sciences showing a connection between, for example,
parental education and children’s education attainment are often considered
sufficient evidence for such a connection. The credibility of the evidence is not
only from the replicated statistical results but also from real-life observation.
Drawing from personal experiences, most people would probably agree that a
higher level of parental education would lead to a higher level of education in
children on average. Such confirmation from life experiences is not available for
interpreting genetic findings. Genotypes are not visible in everyday life. To
develop a credible story that supports statistical findings, other evidence is
needed, such as those from animal studies and biochemical studies. (p. 224; italics
added)

Page 27 of 36

PRINTED FROM OXFORD HANDBOOKS ONLINE (www.oxfordhandbooks.com). © Oxford University Press, 2018. All Rights
Reserved. Under the terms of the licence agreement, an individual user may print out a PDF of a single chapter of a title in
Oxford Handbooks Online for personal use (for details see Privacy Policy and Legal Notice).

Subscriber: Goldsmiths College, University of London; date: 02 February 2019

Designing Multimethod Research

The Narrative Moral of Multimethod Research

Design
We return to the opening of our discussion about multimethod research design, just as
Ulysses eventually returned to Greece after a meandering Odyssey, and that is to stress
the spirit of multimethod research with which we began. True believers of one or another
method, positing different ontological assumptions, will continue to debate the
appropriateness of different methods, as they should in the full spirit of free inquiry. But
in the further spirit of multimethod research such assumptions should be open to
challenge, and a tolerance for entertaining alternative assumptions should be considered.
The spirit of humility, not hubris, and the recognition of limitations in methods are more
likely to advance the cause of science than any authoritative dictates of idealized
methods. Context and contingency apply to research itself, and understanding the
broader social context and the pragmatic contingent decisions made in the conduct of
research must be taken into accounts. What we claim to know is dependent on how we
came to know it. And the multimethod perspective still holds out the promise of closer
approximations to truth about reality or, at a minimum, more moderated contingent
claims to truth that reflect the reality of science itself.

We previously stated our criteria of good scientific research that it be

• Consistent—logically consistent, not random or contradictory

• Corresponding—data and ideas linked in measurement
• Convincing—rhetorically persuasive
• To this we add another—that it be “competent”

Multimethod research is not a magic bullet to truth but a style that still demands rigor
and reflection in addressing the skeptic’s central question: “How do you know?”

Discussion Questions
1. Can one consider mixed methods research a subset of multimethod research?
2. What parts of reality and the world does multimethod research focus on?
3. In what ways is multimethod research frequently superior to single or
monomethod research?
4. Is multimethod research multidisciplinary?
5. What are some classic multimethod studies provided in this chapter, and what
makes them important?

Suggested Websites
http://www.sociology.northwestern.edu/people/faculty/albert-hunter.html

Page 28 of 36

PRINTED FROM OXFORD HANDBOOKS ONLINE (www.oxfordhandbooks.com). © Oxford University Press, 2018. All Rights
Reserved. Under the terms of the licence agreement, an individual user may print out a PDF of a single chapter of a title in
Oxford Handbooks Online for personal use (for details see Privacy Policy and Legal Notice).

Subscriber: Goldsmiths College, University of London; date: 02 February 2019

Designing Multimethod Research

Information about the authors of this chapter.

References
Abbott, A. (1990). Conceptions of time and events in social science methods: Causal and
narrative approaches. Historical Methods: A Journal of Quantitative and Interdisciplinary
History, 23(4), 140–150.

Abbott, A. (1992). From causes to events notes on narrative positivism. Sociological

Methods & Research, 20, 428–455.

Abbott, A. (1995). Sequence analysis: New methods for old ideas. Annual Review of
Sociology, 21, 93–113.

Abbott, A. (1997). Of time and space: The contemporary relevance of the Chicago School.
Social Forces, 75, 1149–1182.

Ajzen, I., & Fishbein, M. (1977). Attitude-behavior relations: A theoretical analysis and
review of empirical research. Psychological Bulletin, 84(5). doi:
10.1037/0033-2909.84.888

Allison, P. D. (1984). Event history analysis: Regression for longitudinal event data.
Newbury Park, CA: Sage.

Anderson, N. (1923). The hobo: The sociology of homeless men. Chicago, IL: University of
Chicago Press.

Baker, S. (1990). Reflection, doubt, and the place of rhetoric in postmodern social theory.
Sociological Theory, 8, 212–245.

Barton, A. H. (1969). Communities in disaster: A sociological analysis of collective stress

situations. Garden City, NJ: Doubleday.

Bennett, J. (1981). Oral history and delinquency: The rhetoric of criminology.

(p. 203)

Chicago, IL: University of Chicago Press.

Blau, P. M., & Duncan, O. D. (1967). American occupational structure. New York, NY: Free
Press.

Bloor, D. (1991). Knowledge and social imagery. Chicago, IL: University of Chicago Press.

Bobo, L., Zubrinsky, C. L., Johnson, J. H. Jr., & Oliver, M. L. (1994). Public opinion before
and after a spring of discontent. In M. Baladassare (Ed.), The Los Angeles riots: Lessons
for the urban future (pp. 103–133). Boulder, CO: Westview.

Booth, W. C. (1983). The rhetoric of fiction. Chicago: University of Chicago Press.

Boswell, J. (1998). Life of Johnson. Edited by R. W. Chapman. Oxford, UK: Oxford

University Press.

Page 29 of 36

PRINTED FROM OXFORD HANDBOOKS ONLINE (www.oxfordhandbooks.com). © Oxford University Press, 2018. All Rights
Reserved. Under the terms of the licence agreement, an individual user may print out a PDF of a single chapter of a title in
Oxford Handbooks Online for personal use (for details see Privacy Policy and Legal Notice).

Subscriber: Goldsmiths College, University of London; date: 02 February 2019

Designing Multimethod Research

Brewer, J., & Hunter, A. (1989). Multimethod research: A synthesis of styles. Newbury
Park, CA: Sage.

Brewer, J., & Hunter, A. (2006). Foundations of multimethod research: Synthesizing

styles. Thousand Oaks, CA: Sage.

Briggs, X. S., Popkin S. J., & Goering J. M. (2010). Moving to opportunity: The story of an
American experiment to fight ghetto poverty. New York, NY: Oxford University Press.

Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the
multitrait-multimethod matrix. Psychological Bulletin, 56, 81.

Campbell, D. T., & Stanley, J. C. (1963). Experimental and quasi-experimental designs for
research. Chicago, IL: Rand McNally.

Cohen-Cole, E., & Fletcher, J. M. (2008). Is obesity contagious? Social networks vs.
environmental factors in the obesity epidemic. Journal of Health Economics, 27, 1382–
1387. doi: 10.1016/j.healeco.2008.04.005

Cook, T. D. (2002). Randomized experiments in educational policy research: A critical

examination of the reasons the educational evaluation community has offered for not
doing them. Educational Evaluation and Policy Analysis, 24(3), 175–199.

Cook, T. D., & Campbell, D. T. (1979). Quasi- experimentation: Design and analysis issues
for field settings. Boston, MA: Houghton Mifflin.

Cook, T. D., Shadish, W. R., & Wong, V. C. (2008). Three conditions under which
experiments and observational studies produce comparable causal estimates: New
findings from within-study comparisons. Journal of Policy Analysis and Management, 27,
724–750. doi: 10.1002/pam.20375

Davis, J. A., Spaeth, J. L., & Huson, C. (1961). A technique for analyzing the effects of
group composition. American Sociological Review, 26, 215–225.

Denzin, N. K. (1970). The research act in sociology: A theoretical introduction to

sociological methods. Piscataway, NJ: Aldine.

DeVault, M. L. (1990). Women write sociology: Rhetorical strategies. In A. Hunter (Ed.),

The rhetoric of social research: Understood and believed, (pp. 97–110.). New Brunswick,
NJ: Rutgers University Press.

Dewey, J. (1938). Logic: The theory of inquiry. New York, NY: Holt, Rinehart and Winston.

Donne, J. (1896). Epigrams. In E. K. Chambers (Ed.) Poems of John Donne (Vol II, p. 210).
London: Lawrence & Bullen.

Duneier, M. (2004). Scrutinizing the heat: On ethnic myths and the importance of shoe
leather. Contemporary Sociology, 2, 139–150.

Page 30 of 36

PRINTED FROM OXFORD HANDBOOKS ONLINE (www.oxfordhandbooks.com). © Oxford University Press, 2018. All Rights
Reserved. Under the terms of the licence agreement, an individual user may print out a PDF of a single chapter of a title in
Oxford Handbooks Online for personal use (for details see Privacy Policy and Legal Notice).

Subscriber: Goldsmiths College, University of London; date: 02 February 2019

Designing Multimethod Research

Duneier, M. (2006). Ethnography, the ecological fallacy, and the 1995 Chicago heat wave.
American Sociological Review, 71, 679–688.

Erikson, K. T. (1976). Everything in its path: Destruction of community in the Buffalo

Creek flood. New York, NY: Simon and Schuster.

Erikson, K. (1994). A new species of trouble: The human experience of modern disasters.
New York, NY: W. W. Norton.

Espeland, W. N., & Stevens, M. L. (1998). Commensuration as a social process. Annual

Review of Sociology, 24, 313–343.

Fine, G.A. (2012). Tiny publics: A theory of group action and culture. Ithaca, NY: CUP
Services.

Form, W. H., & Nosow, S. (1958). Community in disaster. New York, NY: Harper.

Freudenburg, W. R., Gramling, R., Laska, S. & Erikson, K. (2009). Catastrophe in the
making: The engineering of Katrina and the disasters of tomorrow. Washington, DC:
Island Press.

Granovetter, M. (1985). Economic action and social structure: The problem of

embeddedness. American Journal of Sociology, 91, 481–510.

Greene, J. C., Caracelli, V. J., & Graham, W. F. (1989). Toward a conceptual framework for
mixed-method evaluation designs. Educational Evaluation and Policy Analysis, 11, 255–
274.

Guo, G., & Adkins, D. E. (2008). How is a statistical link established between a human
outcome and a genetic variant? Sociological Methods Research, 37, 201–226.

Guo, G., Elder, G. H., Cai, T., & Hamilton, N. (2009). Gene–environment interactions:
Peers’ alcohol use moderates genetic contribution to adolescent drinking behavior. Social
Science Research, 38, 213–224.

Guo, G., Hardie, J. H., Owen, C., Daw, J.K., Fu, Y., Lee, H . . . Duncan, G. (2009). DNA
collection in a randomized social science study of college peer effects. Sociological
Methodology, 39, 1–29.

Gusfield, J. R. (1990). Two genres of sociology: A literary analysis of the American

occupational structure and tally’s corner. In A. Hunter (Ed.), The rhetoric of social
research: Understood and believed, (pp. 62–96.). New Brunswick, NJ: Rutgers University
Press.

Hunter, A. (1986). Love Canal emergency declaration area: Proposed habitability criteria
study: Vol. 1. Technical Review Committee. New York, NY: US Environmental Protection
Agency, US Department of Health and Human Services/Centers for Disease Control, New

Page 31 of 36

PRINTED FROM OXFORD HANDBOOKS ONLINE (www.oxfordhandbooks.com). © Oxford University Press, 2018. All Rights
Reserved. Under the terms of the licence agreement, an individual user may print out a PDF of a single chapter of a title in
Oxford Handbooks Online for personal use (for details see Privacy Policy and Legal Notice).

Subscriber: Goldsmiths College, University of London; date: 02 February 2019

Designing Multimethod Research

York State Department of Health, New York State Department of Environmental

Conservation.

Hunter, A. (Ed.). (1990). The rhetoric of social research: Understood and believed. New
Brunswick, NJ: Rutgers University Press.

Hunter, A., & Maxfield, M. G. (1980). Reactions to Crime Project: Vol. 1. Methodological
overview of the Reactions to Crime Project. Evanston, IL: Northwestern University,
Center for Urban Affairs.

Investopedia. (2014). Revealed preference. Retrieved from http://

www.investopedia.com/terms/r/revealed-preference.asp

Jacobs, J. A. (2005). Multiple methods in ASR. Footnotes: Newsletter of the American

Sociological Review, 33(9). Retrieved from http://www.asanet.orgwww.asanet.org/
footnotes/dec05/indextwo.html.

Jahoda, M, Lazarsfeld, P. F., & Zeisel, H, (1971). Marienthal: The sociography of an

unemployed community. New Brunswick, NJ: Transaction. (Originally published in 1933)

Jankowski, M. S. (1991). Islands in the street: Gangs and American urban society.
Berkeley: University of California.

(p. 204) Joppa, L. N., McInerny, G., Harper, R., Salido, L., Takeda, K., O’Hara, K., . . .
Emmott, S. (2013). Troubling trends in scientific software use. Science, 340, 814–815.

Johnson, B., & Turner, L. A. (2003). Data collection strategies in mixed methods research.
In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social & behavioral
research, (pp. 297–319). Thousand Oaks, CA: Sage.

Kahneman, D. (2011). Thinking, fast and slow. New York, NY: Farrar, Strauss & Giroux.

Kanter, R. M. (1993). Men and women of the corporation. New York, NY: Basic Books.

Kendra, J. M., & Wachtendorf, T. (2003). Elements of resilience after the World Trade
Center disaster: Reconstituting New York City’s Emergency Operations Centre. Disasters,
27(1), 37–53. doi:10.1111/1467-7717.00218

King, G., Keohane, R. O., & Verba, S. (1994). Designing social inquiry: Scientific inference
in qualitative research. Princeton, NJ: Princeton University Press.

Klinenberg, E. (2003). Heat wave: A social autopsy of disaster in Chicago. Chicago, IL:
University of Chicago Press.

Krieger, S. (1983). The mirror dance: Identity in a women’s community. Philadelphia, PA:
Temple University Press.

LaPiere, R. T. (1934). Attitudes vs. actions. Social Forces, 13, 230–237.

Page 32 of 36

PRINTED FROM OXFORD HANDBOOKS ONLINE (www.oxfordhandbooks.com). © Oxford University Press, 2018. All Rights
Reserved. Under the terms of the licence agreement, an individual user may print out a PDF of a single chapter of a title in
Oxford Handbooks Online for personal use (for details see Privacy Policy and Legal Notice).

Subscriber: Goldsmiths College, University of London; date: 02 February 2019

Designing Multimethod Research

Lazarsfeld, P. F. (1945) Foreward to Ernest Greenwood, Experimental Sociology. New

York, NY: King’s Crown Press.

Lever, J. (1978). Sex differences in the complexity of children’s play and games. American
Sociological Review, 43, 471–483.

Liebow, E. (1967). Tally’s corner: A study of Negro streetcorner men. New York, NY:
Rowman & Littlefield.

Mahoney, J., & Rueschemeyer, D. (Eds.). (2003). Comparative historical analysis in the
social sciences. Cambridge, UK: Cambridge University Press.

Manchester, W. (1978). American Caesar: Douglas McArthur. New York, NY: Dell.

McClelland, D. C. (1961). Achieving society. Princeton, NJ: Van Nostrand.

Merton, R. K. (Ed.). (1968). Social theory and social structure. New York, NY: Free Press.

Merton, R. K., & Lazersfeld, P. F. (Eds.). (1950). Continuities in social research: Studies in
the scope and method of “The American Soldier.” New York, NY: Free Press.

Mills, C. W. (1959). The sociological imagination. New York, NY: Oxford University Press.

Moore, B. (1958). Political power and social theory: Six studies. Cambridge, MA: Harvard
University Press.

Moore, B. (1966). Social origins of dictatorship and democracy: Lord and peasant in the
making of the modern world. Boston, MA: Beacon Press.

Morse, J. M. (2003). Principles of mixed methods and multimethod research design. In A.

Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social & behavioral
research (pp. 189–208). Thousand Oaks, CA: Sage.

Mulkay, M. J. (1979). Science and the sociology of knowledge. London: Allen & Unwin.

Office of Income Security Policy, Office of the Assistant Secretary for Planning and
Evaluation, US Department of Health and Human Services. (1983). Overview of the
Seattle–Denver income maintenance experiment final report. Washington, DC: Author.

Pager, D., & Quillian, L. (2005). Walking the talk? What employers say versus what they
do. American Sociological Review, 70, 355–380. doi: 10.1177/000312240507000301

Prince, S. H. (1920). Catastrophe and social change, based upon a sociological study of
the Halifax disaster. Studies in History, Economics, and Public Law 212. New York, NY:
Columbia University Press.

Putnam, R. D. (2001). Bowling alone: The collapse and revival of American community.
New York, NY: Simon and Schuster.

Page 33 of 36

PRINTED FROM OXFORD HANDBOOKS ONLINE (www.oxfordhandbooks.com). © Oxford University Press, 2018. All Rights
Reserved. Under the terms of the licence agreement, an individual user may print out a PDF of a single chapter of a title in
Oxford Handbooks Online for personal use (for details see Privacy Policy and Legal Notice).

Subscriber: Goldsmiths College, University of London; date: 02 February 2019

Designing Multimethod Research

Ragin, C. C. (1987). The comparative method: Moving beyond qualitative and quantitative
strategies. Berkeley: University of California Press.

Ragin, C. C., & Becker, H. S. (Eds.). (1992). What is a case? Exploring the foundations of
social inquiry. Cambridge, UK: Cambridge University Press.

Raudenbush, S. W., & Bryk, A. S. (2001). Hierarchical linear models: Applications and
data analysis methods. Advanced Quantitative Techniques in the Social Sciences 1.
Thousand Oaks, CA: Sage.

Rossi, P. H., Lipsey, M. W., & Freeman, H. E. (2004). Evaluation: A systematic approach.
Thousand Oaks, CA: Sage.

Saey, T. H. (2013, June 29). Gene weakly linked to education level. Science News.
Retrieved from https://www.sciencenews.org/article/genes-weakly-linked-
education-level

Sampson, R. J., Morenoff, J. D., & Gannon-Rowley, T. (2002). Assessing “neighborhood

effects”: Social processes and new directions in research. Annual Review of Sociology, 28,
443–478.

Samuelson, P. (1948). Economics. New York, NY: McGraw-Hill.

Sandelowski, M. (2003). Tables or tableaux? The challenges of writing and reading mixed
methods studies. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in
social & behavioral research, (pp. 321–350). Thousand Oaks, CA: Sage.

Scott, W. R., & Meyer, J. W. (Eds.). (1994). Institutional environments and organizations:
Structural complexity and individualism. Thousand Oaks, CA: Sage.

Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-
experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin.

Shaw, C. R. (2013). The jack-roller. Eastford, CT: Martino. (Originally published in 1930)

Shea, C. (2009). The nature–nurture debate, redux: Genetic research finally makes its
way into the thinking of sociologists. The Chronicle of Higher Education, 54(31), B6.

Shils, E. (1957). Primordial, personal, sacred and civil ties: Some particular observations
on the relationships of sociological research and theory. The British Journal of Sociology,
8(2), 130–145.

Simmel, G. (1971). On individuality and social forms. Chicago, IL: University of Chicago
Press.

Skocpol, T (Ed.). (1984).Vision and method in historical sociology. New York, NY:
Cambridge University Press.

Page 34 of 36

PRINTED FROM OXFORD HANDBOOKS ONLINE (www.oxfordhandbooks.com). © Oxford University Press, 2018. All Rights
Reserved. Under the terms of the licence agreement, an individual user may print out a PDF of a single chapter of a title in
Oxford Handbooks Online for personal use (for details see Privacy Policy and Legal Notice).

Subscriber: Goldsmiths College, University of London; date: 02 February 2019

Designing Multimethod Research

Skocpol, T., & Somers, M. (1980). The uses of comparative history in macrosocial inquiry.
Comparative Studies in Society and History, 22(2), 174–197.

Small, M. L. (2011). How to conduct a mixed methods study: Recent trends in a rapidly
growing literature. Annual Review of Sociology, 37, 57–86. doi: 10.1146/annurev.soc.
012809.102657

Stallings, R. A. (1997). Sociological theories and disaster studies. Newark, DE: Disaster
Research Center.

Stouffer, S. A., Lumsdaine, A. A., Lumsdaine, M. H., Williams R. M. Jr., Smith, M. B., Janis,
I. L., . . . & Cottrell L. S. Jr. (p. 205) (1949). Studies in social psychology in World War II:
Vol. 2. The American soldier: Combat and its aftermath. Oxford, UK: Princeton University
Press.

Sutherland, E. H. (Ed.) (1937). The professional thief. Chicago, IL: University of Chicago
Press

Thomas, W. I. (1923). The unadjusted girl. Oxford, UK: Little, Brown and Co.

Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases.
Science, 185, 1124–1131.

Uzzi, B. (1999). Embeddedness in the making of financial capital: How social relations
and networks benefit firms seeking financing. American Sociological Review, 64, 481–
505.

Wade, H. D. (2004). On operationalisms and economics. Journal of Economic Issues, 38(4),

953–968.

Wallerstein, I. M. (1974). The modern world-system: Vol. 1. Capitalist agriculture and the
origins of the European world-economy in the sixteenth century. New York, NY: Academic
Press.

Wallerstein, I. M. (2004). World-systems analysis: An introduction. Durham, NC: Duke

University Press.

Walton, J., & Ragin, C. (1990). Global and national sources of political protest: Third
world responses to the debt crisis. American Sociological Review, 55, 876–890.

Weber, M. (1905). The protestant ethic and the spirit of capitalism. New York, NY:
Routledge. 2001.

Albert Hunter

Albert Hunter is currently professor of sociology at Northwestern University and

Director of the Urban Studies Program where he is affiliated with the Institute for
Policy Research and the Transportation Center. His degrees are from Cornell (BA)

Page 35 of 36

PRINTED FROM OXFORD HANDBOOKS ONLINE (www.oxfordhandbooks.com). © Oxford University Press, 2018. All Rights
Reserved. Under the terms of the licence agreement, an individual user may print out a PDF of a single chapter of a title in
Oxford Handbooks Online for personal use (for details see Privacy Policy and Legal Notice).

Subscriber: Goldsmiths College, University of London; date: 02 February 2019

Designing Multimethod Research

and the University of Chicago (MA,PhD). He has previously taught at the University
of Chicago, Wesleyan University, the University of Rochester and had visiting
appointments at The London School of Economics, Yale University, the University of
Edinburgh, and the University of Paris VIII-St. Denis. He is the author of numerous
articles and books and most of his research is multimethod including two major
multi-institutional multi-site collective projects conducted at the Center for Urban
Affairs and Policy research – The Reactions to Crime Project, and the Changing
Relations Project on Ethnic Diversity. His books include Symbolic Communities: The
Persistence and Change of Chicago’s Local Communities (1974) two books on
“multimethod research” (with John Brewer), Multimethod Research: A Synthesis of
Styles (1989) and Foundations of Multimethod Research (2006); a book on The
Rhetoric of Social Research: Understood and Believed (1990), and most recently a
book on “civil society” titled Pragmatic Liberalism: Constructing a Civil Society (with
Carl Milofsky, 2007). He continues research on the “symbolic ecology” of cities
including a comparative study of neighborhood response to gangs, and a comparative
study of elite and poor suburbs of Chicago. He is also engaged in a ongoing
comparative study of civil society in the US and the UK. He has served as Editor of
Urban Affairs Review, Chair of the Community Section of the ASA, and Chair of the
Plan Commission of the City of Evanston.

John D. Brewer

John D. Brewer is Professor of Sociology Emeritus at Trinity College in Hartford

Connecticut. Previously he taught at the University of California, Los Angeles, York
University in Toronto, Canada, and Wesleyan University in Middletown, Connecticut
and has held a visiting appointment at the University of Connecticut School of Law.
He received his BA,. MA, and Ph.D degrees from the University of Chicago, where he
was a Woodrow Wilson Fellow. He has studied formal organizations, written about
problems in organization research and theory, and served in the elected position of
Secretary to the American Sociological Association’s Section on Organizations and
Occupations. He is co-author (with Albert Hunter, SAGE, 1989) Multimethod
Research: A Synthesis of Styles, and co-author (with Albert Hunter, SAGE, 2006)
Foundations of Multmethod Research: Synthesizing Styles.

Page 36 of 36

PRINTED FROM OXFORD HANDBOOKS ONLINE (www.oxfordhandbooks.com). © Oxford University Press, 2018. All Rights
Reserved. Under the terms of the licence agreement, an individual user may print out a PDF of a single chapter of a title in
Oxford Handbooks Online for personal use (for details see Privacy Policy and Legal Notice).

Subscriber: Goldsmiths College, University of London; date: 02 February 2019