Scribd
Upload
69 views8 pages

2specifying The Problem and Retrieving Studies

This document discusses how to conduct a meta-analysis, including specifying the problem, retrieving relevant studies, determining when a meta-analysis is applicable, calculating effect sizes, developing inclusion/exclusion criteria, and ensuring a comprehensive search for studies. Key recommendations include thoroughly reviewing the literature on the topic, deciding on clear objectives, developing a theoretical framework, and establishing explicit criteria for including studies. The document emphasizes calculating standardized effect sizes to allow direct comparison of findings across studies.

Uploaded by

Đinh Đức Tâm
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
69 views8 pages

2specifying The Problem and Retrieving Studies

This document discusses how to conduct a meta-analysis, including specifying the problem, retrieving relevant studies, determining when a meta-analysis is applicable, calculating effect sizes, developing inclusion/exclusion criteria, and ensuring a comprehensive search for studies. Key recommendations include thoroughly reviewing the literature on the topic, deciding on clear objectives, developing a theoretical framework, and establishing explicit criteria for including studies. The document emphasizes calculating standardized effect sizes to allow direct comparison of findings across studies.

Uploaded by

Đinh Đức Tâm
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 8

Introduction to Meta-Analysis

How to Specify the Problem and Retrieve


Studies?

Ahmet H. Kirca, Ph.D.


Associate Professor
Michigan State University

Discussion

1
When Can You Do a Meta-Analysis?

Meta-analysis is applicable to collections of


research that
Are empirical, rather than theoretical
Produce quantitative results, rather than qualitative
findings
Examine the same constructs and relationships
Have findings that can be configured in a comparable
statistical form (e.g., as effect sizes, correlation
coefficients, odds-ratios, proportions)
Are comparable given the question at hand

Source: Lipsey and Wilson (2001), Practical Meta-Analysis by Sage


3

Forms of Research Findings Suitable to


Meta-Analysis

Pre-post contrasts
Growth rates
Group contrasts
Experimentally created groups
Comparison of outcomes between treatment and comparison
groups
Naturally occurring groups
Comparison of spatial abilities between boys and girls
Rates of morbidity among high and low risk groups
Association between variables
Correlation between personality constructs
Correlation between organizational characteristics/strategies
and outcome varaibles (e.g., performance)
Source: Lipsey and Wilson (2001), Practical Meta-Analysis by Sage
4

2
Recommendations for the Problem Formulation
Stage

Read, Read, and Read


Read key empirical and conceptual articles on the topic
Read qualitative and quantitative reviews related to the
topic
Identify important common variables, study and
measurement characteristics
Watch out for contradictory findings
Decide on the specific objective of the meta-analysis
Develop a preliminary theoretical framework

Discussion

3
Effect Size: The Key to Meta-Analysis

The effect size makes meta-analysis


possible
It is the dependent variable
It standardizes findings across studies such
that they can be directly compared

Source: Lipsey and Wilson (2001), Practical Meta-Analysis by Sage


7

Effect Size: The Key to Meta-Analysis

Any standardized index can be an effect size (e.g.,


standardized mean difference, correlation coefficient,
odds-ratio) as long as it meets the following
Is comparable across studies (generally requires standardization)
Represents the magnitude and direction of the relationship of
interest
Is independent of sample size
Different meta-analyses may use different effect size
indices

Source: Lipsey and Wilson (2001), Practical Meta-Analysis by Sage


8

4
Which Studies to Include?
Have an explicit inclusion and exclusion criteria
The broader the research domain, the more detailed they tend to
become
Refine criteria as you interact with the literature
Components of a detailed criteria
distinguishing features
research respondents
key variables
research methods
cultural and linguistic range
time frame
publication types

Source: Lipsey and Wilson (2001), Practical Meta-Analysis by Sage


9

Recommendations for the Study


Retrieval Stage
Conduct key-word searches of electronic databases using a few
keywords too many keywords ???
Search for references of key studies (i.e., highly cited key publications
or major qualitative reviews)
Conduct manual searches of relevant journals.
Solicit studies and working papers from known authors
Post requests for unpublished studies and working papers on
academic list-servers after you are done with data collection
Be very specific about the screening criteria and eliminate studies
accordingly
Track everything!

5
Example

The Replication Continuum

Pure Conceptual
Replications Replications
You must be able to argue that the collection of studies you are
meta-analyzing examine the same relationship. This may be at
a broad level of abstraction, such as the relationship between
criminal justice interventions and recidivism or between school-
based prevention programs and problem behavior. Alternatively
it may be at a narrow level of abstraction and represent pure
replications.

The closer to pure replications your collection of studies, the


easier it is to argue comparability.

12

6
Methodological Quality Dilemma

Include or exclude low quality studies?


The findings of all studies are potentially in error (methodological
quality is a continuum, not a dichotomy)
Being too restrictive may restrict ability to generalize
Being too inclusive may weaken the confidence that can be
placed in the findings
Methodological quality is often in the eye-of-the-beholder
You must strike a balance that is appropriate to your research
question

Source: Lipsey and Wilson (2001), Practical Meta-Analysis by Sage


13

Searching Far and Wide


The we only included published studies because they
have been peer-reviewed argument
Significant findings are more likely to be published than
nonsignificant findings
Critical to try to identify and retrieve all studies that meet
your eligibility criteria
Potential sources for identification of documents
Computerized bibliographic databases
Authors working in the research domain (email Listserv?)
Conference programs, Dissertations
Review articles
Manual searches of relevant journals
Government reports, bibliographies, clearinghouses
Source: Lipsey and Wilson (2001), Practical Meta-Analysis by Sage
14

7
PRISMA flowchart

Source:
Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group (2009) Preferred Reporting Items for Systematic Reviews and Meta-
Analysis: The PRISMA Statement. PLoS Med 6(7)
For more information, visit www.prisma-statement.org

Discussion

You might also like