Reviewing Mixed Methods Applications / 31

Reviewing Mixed Methods Applications

❖❖ NIH Review Criteria: Applications submitted to the NIH for awards or cooperative agreements to support
biomedical and behavioral research are evaluated for scientific and technical merit through the NIH peer review
system. Reviewers consider criteria as summarized across Center grants, R, F, K, T, P grants (see the NIH website for
details). As discussed in the section on Beyond the R Series – High Quality Mixed Methods Activities in Successful
Fellowship, Career, Training, and Center Grant Applications, the criteria for evaluating grants in each of these
categories varies. However, several general comments about reviewing applications that include mixed methods
research can be made.

❖❖ Use both quantitative and qualitative criteria: In all mixed methods investigations, the criteria for a rigorous
quantitative investigation and a qualitative investigation should be met in evaluating a mixed methods investigation
(see The nature of qualitative research and its evidence and The nature of quantitative research and its evidence).
Further, specific criteria about mixed methods need to be employed.

❖❖ Use mixed methods criteria: A body of literature is emerging about how to assess the quality of a mixed methods
investigation. This literature addresses criteria that might be used, what components might go into a mixed methods
study, where to examine a project for mixed methods components, and checklists for assessing the value of such an
investigation. For example, the NSF evaluation guidelines advance quality criteria for quantitative and qualitative
data and methods (Frechtling, 2002), the Robert Wood Johnson website suggests specific guidelines for qualitative
research (Cohen & Crabtree, 2008), and the article by Levin et al. (1997) state guidelines for quantitative research.
The Journal of Mixed Methods Research (Tashakkori & Creswell, 2007) advances criteria for evaluating mixed methods
submissions to the journal.

❖❖ Consult existing mixed methods research quality criteria: Several standards for reviewing the quality of mixed
methods research have been advanced in the literature (see O’Cathain, 2010). Three recent discussions (Creswell
& Plano Clark, 2011; O’Cathain, Murphy, & Nicholl, 2008; Schifferdecker & Reed, 2008) use different criteria: a
methods orientation, a research process orientation, and the timing of phases of the investigation orientation.

• Creswell and Plano Clark (2011) suggest that to evaluate a mixed methods study, the researcher needs to:

–– collect both quantitative and qualitative data;

–– employ rigorous procedures in the methods of data collection and analysis;

–– integrate or mix (merge, embed, or connect) the two sources of data so that their combined use provides a
better understanding of the research problem than one source or the other;

–– use a mixed methods research design and integrate all features of the study with the design; and

–– convey research terms consistent with those being used in the mixed method field.

• O’Cathain, Murphy, and Nicholl (2008) offer guidance for Good Reporting of a Mixed Methods Study
(GRAMMS):

–– describe the justification for using a mixed methods approach to the research question;

–– describe the design in terms of the purpose, priority, and sequence of methods;

–– describe each method in terms of sampling, data collection and analysis;

–– describe where integration has occurred, how it has occurred, and who has participated in it;

–– describe any limitation of one method associated with the presence of the other method; and

–– describe any insights gained from mixing or integrating methods.

• Schifferdecker and Reed (2008) made seven recommendations for steps to be taken in designing, analyzing, and
publishing mixed methods studies:

–– Identify the study design as mixed methods.

–– Decide on the prominence of each data type in data collection, analysis, and results.

Best Practices for Mixed Methods Research in the Health Sciences

 Reviewing Mixed Methods Applications / 32

–– Develop sampling strategies that provide adequate data and adhere to guidelines within the methods
chosen.

–– Decide how and when data are collected, analyzed, and integrated.

–– Set realistic time requirements for each project phase.

–– Explore software programming tools or methods to integrate the quantitative and qualitative data.

–– Review mixed methods research articles to generate ideas for reporting results and displaying data.

A Checklist for Reviewing NIH Mixed Methods Applications: In view of these alternative sets of criteria, we suggest
a checklist for an R Series application that reviewers might use to review the research plan for NIH applications that
incorporate mixed methods research. These sections may be weighted differently and given different emphases depending
on the type of R Series application. Different criteria will apply to other types of applications (e.g., K Series applications,
Center applications) as well as the announced grant mechanism and funding opportunity [see Table 1. Sample review
criteria and strategies for reviewing an R-series application].

Table 1. Sample review criteria and strategies for reviewing an R Series application

Criterion Strategies for Meeting the Criterion NIH Scoring

Significance 1-9

• Does the application make a convincing case that the problem is relevant
(e.g., if aims are achieved, the work will improve knowledge or practice)?

• Can the problem be best studied through the multiple perspectives of mixed
methods research?

Investigator(s) 1-9

• Do the investigator(s) have the required skills to conduct all proposed

methods (e.g., investigator(s) have prior publications and/or grants related
to proposed qualitative, quantitative, and mixed methods; co-investigators
with appropriate expertise are identified to lead each method as needed)?

• Is there evidence that the project leadership is committed to mixed methods

research (e.g., each component of the study is addressed sufficiently and
consistently throughout the application; there are references to current
relevant literature on mixed methods; investigators have experiences
in professional development in mixed methods)?

• Has the approach to collaboration been described (e.g., frequency of

meetings between leaders of different components, management of
differences between co-investigators)?

Innovation 1-9

• Does the use of mixed methods provide a platform for innovative

investigation of the research problem(s) (e.g., provides insights into
mechanisms of organizational change not possible with a single method)?

• Is the combination of methods used innovative, or the way in which they

are integrated innovative?

Best Practices for Mixed Methods Research in the Health Sciences

 Reviewing Mixed Methods Applications / 33

Table 1. Sample review criteria and strategies for reviewing an R Series application

Criterion Strategies for Meeting the Criterion NIH Scoring

Approach 1-9

• Is there a description of the philosophy or theory informing the research

and the ways this philosophy or theory shapes the investigation?

• Have the applicants offered a convincing explanation of why mixed

methods research is needed to address the study aims and the value added
by using this approach (e.g., explained how alternative designs would
be inappropriate or inadequate)?

• Is there a clear description of the full study design, including where

integration occurs (e.g., using a comprehensive figure or matrix)?

• Is the integration of the methods well described, including the timing,

techniques, and responsibilities for integration?

• Is the design appropriate for the study aims?

• Are the methods consistent with established standards of rigor for

quantitative and qualitative data collection and analysis (e.g., sampling,
sample size and analysis plans are specified for each method, with
appropriate citations)?

• Will appropriate computer software be used for each analytic component,

and if not, is a convincing rationale provided?

• Is the study feasible within its proposed time frame and resources
(e.g., a timetable is provided that allocates time for data integration)?

Environment 1-9

• Is there evidence that the institution supports mixed methods research

(e.g., forums for multidisciplinary collaborations, faculty with funding
for mixed methods research)?

Rating Scale: See the general NIH guidelines for scoring for more information about the evaluation categories and
scoring.

Key References and Resources

❖❖ Cohen, D., & Crabtree, B. (2008). Robert Wood Johnson, qualitative research guidelines project. Retrieved from
http://www.qualres.org/.

❖❖ Creswell, J. W., & Plano Clark, V. L. (2011). Designing and conducting mixed methods research (2nd ed.). Thousand
Oaks, CA: Sage.

❖❖ Frechtling, J. (January, 2002). The 2002 user friendly handbook for project evaluation. Washington D.C.: National
Science Foundation. Retrieved from http://www.nsf.gov/pubs/2002/nsf02057/start.htm.

❖❖ Levin J. S., Glass, T. A., Kushi, L. H., Schuck, J. R., Steele, L., & Jonas, W. B. (1997). Quantitative methods in
research on complementary and alternative medicine, Medical Care, 35, 1079-1094.

Mixed Methods Research in the Health Sciences

 Reviewing Mixed Methods Applications / 34

❖❖ O’Cathain, A. (2010). Assessing the quality of mixed methods research: Toward a comprehensive framework. In A.
Tashakkori & C. Teddlie (Eds.), SAGE handbook of mixed methods in social & behavioral research (2nd ed.) (pp. 531-
555) Thousand Oaks, CA: Sage.

❖❖ O’Cathain, A., Murphy, E., & Nicholl, J. (2008). The quality of mixed methods studies in health services research.
Journal of Health Services Research Policy, 13(2), 92-98.

❖❖ Plano Clark, V. L., & Creswell, J. W. (2010). Understanding research: a consumer’s guide. Upper Saddle River, N.J.:
Prentice Hall – Merrill.

❖❖ Schifferdecker, K. D. & Reed, V.A. (2009). Using mixed methods research in medical education: Basic guidelines for
researchers. Medical Education, 43, 637-644.

❖❖ Tashakkori, A., & Creswell, J. W. (2007). The new era of mixed methods. [Editorial]. Journal of Mixed Methods
Research, 1(1), 3-7.