Quality frameworks & procedural

checklists for mixed methods

research
Dr Roslyn Cameron
AVETRA 2011

AIM
Overview of research quality criteria in
quantitative and qualitative research before
presenting a synthesis of quality frameworks and
procedural checklists emerging from the field of
mixed methods research.
Procedural checklists and quality frameworks are
useful pedagogic tools for ECRs
Helpful to established researchers: teach
research methods; examine theses; review
papers etc...

What constitutes good research?

The concept of rigour is often referred to along
with theoretical and methodological robustness
when reference is made to making some form of
evaluation or critique of research as process (act)
and research as product (publication).
Andrews and Halcomb (2009, p. xvi) define rigor
as The thoroughness, accuracy, confirmability
and ethical soundness of all aspects of a studys
design.

This paper.....
The potential utilisation of quality criteria for 3 sets of stakeholders in the VET
research community:
higher degree research students;
early career and established researchers
research funding bodies.
Brief comparison of the criteria for evaluating research funding proposals from three
VET relevant funding bodies:
NSW Department of Education and Training (DET)
National Centre for Vocational Education Research (NCVER)
Australian Research Council (ARC).
Commonly agreed quality criteria of quantitative (QUANT) research
Multiple stances taken by qualitative (QUAL) researchers in terms of quality criteria
Quality criteria and procedural checklists mixed methods research (MMR)

Use of Quality Frameworks: HDR Candidates

Guidelines and standards for the design,
conduct and reporting of research (process)
Guidelines for critiquing research (process) &
literature (product)
Tool for reflexivity-critical self reflection
Guidance planning & production of research
products (conference papers; articles;
dissertations; reports)

Use of Quality Frameworks: ECR & established

researchers
Pedagogic tool for teaching research
methodology
Assist in roles such as: reviewing conference
papers, journal articles and research reports
Guide to thesis examination
Self reflective tool for own research and research
reporting
Assist in writing & judging research grant
applications

Use of Quality Frameworks: Research funding

bodies
Writing research funding grant
descriptions/selection criteria/call for tenders
Assessments of research proposals and
funding applications
Evaluate research reports and publications
Judge the use/practical value/ relevance of
research

Criteria for evaluating research funding

proposals
3 VET relevant funding bodies:
NSW Department of Education and Training
(DET)
National Centre for Vocational Education
Research (NCVER)
Australian Research Council (ARC).

NSW DET
Criteria for Quality Research, in which the
department gives equal value to both
quantitative and qualitative methods and has
listed criteria for judging the quality of research
under two main areas:
Methodological and theoretical robustness
12 main dot points
Value and impact
3 main dot points

NVETRE
Table 1: NVETRE Funding Criteria

Essential Criteria
Desirable Criteria
1. Proposed research program
1. Research team composition and skills
2. Research questions, methodology and
2. Value adding and dissemination
timeframe
3. Data analysis skills for research
3. Research experience, expertise &
programs proposing a large
related research
quantitative component
4. Project quality assurance and risk
management
5. Value for money
Source: NCVER (2010)

ARC
Table 2: Selection Criteria for ARC Linkage Project proposals commencing in 2011

Criteria
Investigators

Description
a. Research opportunity and performance evidence.

Weighting
20%

b.Capacity to undertake and manage the proposed

research.
Proposed project
content

Nature of the
alliance,
commitment from
Partner
Organisation(s)
and Budget

a. Significance and innovation

(25%)

b.Approach and Training

(20%)

c. National Benefit

(10%)

a. Is there evidence that each of the Partner

Organisation(s) is genuinely committed to, and
prepared to collaborate in, the research project?
b.Will the proposed research encourage and develop
strategic research alliances between the higher
education organisation(s) and other organisation(s)?
c. Value for money and budget justification.

Source: ARC (2010)

55%

25%

HISTORICAL DEVELOPMENT OF
QUALITY CRITERIA

QUANT

QUAL

MMR

Validity
Reliability
Replicability
Generalisability

3 stances:

Beyond
Triangulation

1. Use QUANT
criteria
2. Use QUAL criteria
3. Cant use pre
determined criteria
for QUAL research

Sale & Brazil (2004)

GRAMMS (2008)

Quality criteria in quantitative research

Majority of the discussion on quality frameworks in
quantitative research is implicit, rather than
explicit and is often referred to in the products of
research as part of the stages of the research
process (e.g, sampling and measures).
Most research methods textbooks will refer to the
concepts of validity and reliability which are
rooted in the positivist and quantitative traditions
of scientific method

Commonly agreed quality criteria in quantitative

research
Table 3: Quality criteria for judging quantitative research

Criteria
Validity

Description
The degree to which a research tool measures what it is supposed to
measure
Reliability
The degree of consistency with which a research tool measures what it is
supposed to measure
Replicability
The same interpretation will be drawn if the study is repeated by
different researchers with different respondents following the same
methods
Generalisability The degree to which we can infer the findings from the research sample
to the population
Source: Andrews and Halcomb (2009)

Quality criteria in qualitative research

...the rise of qualitative research over the last
25-30 years represents one of the reasons for
the growing interest in research quality criteria
because it is widely assumed that whereas
quality criteria for quantitative research are
well known and widely agreed, that is not the
case for qualitative research
(Bryman, Becker and Sempik 2008, p. 262).

3 broad stances for judging qualitative research:

QUAL research should be judged according to
the same criteria as quantitative research
(reliability & validity)
QUAL research should be judged using its
own criteria (Lincoln and Guba 1985)
Appropriateness of any predetermined criteria
for judging qualitative research is questioned
(Rolfe, 2006; Sandelowski & Barroso, 2002).

First stance Johnson (1997)

In qualitative research 3 types of validity can be
discussed:
1. Descriptive validity-factual accuracy of the account
as reported by the qualitative researcher
2. Interpretive validity- the degree that the participants
viewpoints, thoughts, intentions, and experiences are
accurately understood and reported by the qualitative
researcher
3. Theoretical validity: the degree that a theory or
theoretical explanation developed from a research
study fits the data and is, therefore, credible and
defensible

Arguments against applying traditional QUANT

criteria to QUAL research
Scientific discipline or rigor is valued because it is
associated with the worth of research outcomes
and studies are critiqued as a means of judging
rigor. Qualitative research methods have been
criticized for lack of rigor. However, these
criticisms have occurred because of attempts to
judge the rigor of qualitative studies using rules
developed to judge quantitative studies. Rigor
needs to be defined differently for qualitative
research since the desired outcome is different
(Burns & Grove, 2005, p. 55)

Second stance: Common criteria for QUAL

Definition of trustworthiness the degree of confidence that the
researcher has that their qualitative data and findings are credible,
transferable and dependable (Andrews and Halcomb 2009, p. xvii)
Trustworthiness was a term proposed by Lincoln and Guba (1985)
and is often referred to as a goodness of fit criteria which parallels
the term rigor in quantitative research.
A set of four criteria to determine the trustworthiness of QUAL
research:
Credibility
Transferability
Dependability
Confirmability.

4 criteria of Trustworthiness
Credibility (in preference to internal validity) is one of the most important
factors in establishing trustworthiness and is about determining how
congruent the findings are with reality.
Transferability (in preference to external validity/generalisability) requires
the researcher to provide sufficient data and context to enable the
audience to judge whether the findings can be applied to other situations
and contexts.
Dependability (in preference to reliability) refers to having sufficient
details and documentation of the methods employed so that the study can
be scrutinised and replicated.
Confirmability (in preference to objectivity) refers to ensuring that the
studys findings are the result of the experiences of the informants rather
than the preferences of the researcher(s) and can be achieved through an
audit trail of the raw data, memos, notes, data reduction and analysis.

4 criteria of Trustworthiness
Table 4: Quality Criteria for Qualitative Research

Credibility
Prolonged
engagement of site
Persistent
observation

Transferability
Identical elements
Theoretical/
purposive sampling
Thick description

Peer briefing
Triangulation
Member checks
Source: Guba and Lincoln (1985)

Dependability
Multiple data
collection methodstriangulation

Confirmability
Use triangulation
Practice reflexivity
Confirmability audit
through member
checking

Some QUAL research developed own criteria

Grounded theory
Charmaz (2006) proposes four quality
criteria for judging grounded theory:
credibility
originality
resonance
usefulness

Third stance
Sandelowski and Barroso (2002) and Rolfe (2006)
question the appropriateness of any predetermined
criteria for judging qualitative research as there is no
unified qualitative research paradigm.
We need to either acknowledge that the commonly
perceived quantitative-qualitative dichotomy is in fact a
continuum which requires a continuum of quality
criteria, or to recognize that each study is individual
and unique, and that the task of producing frameworks
and predetermined criteria for assessing the quality of
research studies is futile (Rolfe, 2006, p. 304).

Mixed methods research (MMR)

Teddlie and Tashakkori (2010, p. 5) define mixed
methods research (MMR) as:
The broad inquiry logic that guides the selection of
specific methods and that is informed by conceptual
positions common to mixed methods practitioners
(e.g., the rejection of either-or choices at all levels of
the research process). For us, this definition of
methodology distinguishes the MMR approach to
conducting research from that practiced in either the
QUAN or QUAL approach.

Creswell and Plano Clark (2007) time periods in

the evolution of MMR
Formative period (1950s - 1980s)
Paradigm debate period (1970s late 1990s)
Procedural development period
(late 1980s 2000)
Advocacy as a separate design
period (2000+).

Evolution of MMR CRITERIA

Triangulation

Sale & Brazil

(2004)

Use of
triangulation in
QUAL to meet
quality criteria
Critiques: linear
and one
dimensional

Truth Value
Applicability
Consistency
Neutrality

OCathain,
Murphy &
Nicholl (2008)
GRAMMS
6 items

MMR- Sale and Brazil (2004)

Truth value
(QUAL-Credibility vs. QUANT-Internal validity)
Applicability
(QUAL-Transferability vs. QUANT-External validity)
Consistency
(QUAL-Dependability vs. QUANT-Reliability)
Neutrality
(QUAL-Confirmability vs. QUANT-Objectivity)

Sale & Brazil (2004)

QUANT

QUAL

MMR

Internal validity

Credibility

Truth value

External validity/
generalisability

Transferability/
Fittingness

Applicability

Reliability

Dependability

Consistency

Objectivity

Confirmability

Neutrality

Cooksey (2008)
In the social and behavioural sciences, there is a
continuing debate about the criteria one should use to
judge the research quality, impact and contribution.
The crux of this debate has centred on the different
meanings held for the criteria of validity and
generalisability within various research traditions or
paradigms Meanings of internal validity and
external validity, two of the dominant criteria in the
positivistic or normative paradigm, have been
borrowed, distorted and recast to fit different
expectations and paradigm assumptions
(Cooksey 2008, p. 4)

GRAMMS-OCathain, Murphy & Nicholl (2008)

1. Describe the justification for using a mixed methods
approach to the research question
2. Describe the design in terms of the purpose, priority and
sequence of methods
3. Describe each method in terms of sampling, data
collection and analysis
4. Describe where integration has occurred, how it has
occurred and who has participated in it
5. Describe any limitation of one method associated with the
presence of the other method
6. Describe any insights gained from mixing or integrating
methods

Procedural checklists for MMR

Table 6: Procedural checklists for mixed methods research

Collins and OCathain

Research Formulation Phase:
1. Importance of a definition
2. Importance of a mental model for
mixing
3. Utilizing typologies of designs
4. Selecting the reason, rationale, and
purpose for mixing
5. Determining the research question
Research Planning Phase:
6. Selecting a mixed methods design
7. Determining the sampling design
Research Implementation Phase:
8. Collecting data
9. Conducting data analysis
10. Legitimating inferences and
formulating generalizations

Andrews and Halcomb

Planning a mixed methods study:
Purpose and relevance
Theoretical orientation
Research questions

Sampling strategy
Methods of investigation

Methods of analysis

Sources: Collins and OCathain (2009, p.2-6) and Andrews and Halcomb (2009, p. 35)

Conclusion-key messages
Several approaches to addressing the quality of research and
quality criteria can range from commonly agreed to sets of criteria
for mono-method quantitative positivist traditions, to a much more
contested terrain within qualitative research.
The manner in which research funding bodies assess quality also
ranges and has been noted.
3 main stances taken in qualitative research and hinted at quality
criteria that has been developed for specific qualitative
methodologies (e.g, for grounded theory).
Mixed methods is a relatively recent and emerging movement and
yet members of the mixed methods research community have
begun to develop quality criteria and frameworks to enable the
evaluation of a mixed methods study in terms of process (act) and
product (publication).

Conclusion-key messages
Those engaged in the teaching of research methods
and/or of building research capacity need to be
become familiar with the emerging mixed methods
movement and its associated theoretical
underpinnings, designs, nomenclature and the quality
frameworks and criteria that is being developed within.
Cameron (2010) methodological scan of VET based
research, the most dominant approaches were
qualitative (45%) and mixed methods (15%). This is
evidence in itself of the need to embed quality
frameworks and criteria into research training and
capacity building.

Conclusion-key messages
VET researchers need to be aware of this array
of quality criteria and they need to acknowledge
this when choosing and arguing for a set of
criteria that they apply to their own research
Those in charge with building research capacity
in the VET research community be cognisant of
this array of criteria and the need to impart this
knowledge to novice VET researchers.

Dr Roslyn Cameron
r.cameron@cqu.edu.au