Ir J Med Sci (2018) 187:447–452

DOI 10.1007/s11845-017-1663-3

HISTORICAL AND LITERARY

Step by step guide to do a systematic review and meta-analysis

for medical professionals
Y. Bashir 1 & K. C. Conlon 1

Received: 4 July 2017 / Accepted: 10 July 2017 / Published online: 22 July 2017
# Royal Academy of Medicine in Ireland 2017

Abstract new publications that come out each year. Systematic review
Introduction Systematic review and meta-analysis are statis- and meta-analysis provide qualitative and quantitative synthesis
tical tools used to review researches performed on a same of the evidence on an aspect of a particular topic under consid-
topic. They extract the collective effect of the studies per- eration, hence making it easier for medical professionals to
formed on the topic of interest after statistically analysing make appropriate decisions for their patients without being
the data of all the studies included. misdirected by any inappropriately designed or biased study.
Aims and objectives Systematic reviews and meta-analysis According to Cochrane, systematic review summarises the
are getting more and more popular in the medical field. results of all available carefully designed healthcare studies and
Statistics is never the strong aspect of medical professionals, provides a high level of evidence on the effectiveness of
and facing a large number of statistical tests and values could healthcare interventions. Meta-analysis is a statistical tool for
be quite confusing for them. The aim of this article is to sim- estimating the mean and variance of underlying population ef-
plify these two very important research modalities for medical fects from a collection of empirical studies [1]. For most medical
professionals. professionals, systematic review and meta-analysis look like a
Conclusion This article will provide a step-to-step guide for creation from the special gizmos of statisticians, and they would
the medical colleagues to perform a meta-analysis if they are not even think about conducting one. On the other hand, others
interested. think it is a very straightforward task that merely involves
analysing work of other researchers; however, once they have
Keywords Education . Guide . Meta-analysis . Research . started one, they soon realise the magnitude of the task they have
Systematic review committed to. Many systematic reviews are abandoned as unfin-
ished projects. The aim of this paper is to simplify the process for
medical professionals. It will require you to get help and guidance
from other professionals including librarians and statisticians.
Introduction
Figure 1 shows the hierarchy of evidence [2] and gives an
idea of the value of meta-analysis in the research methodolo-
The field of Medicine evolved enormously during the twentieth
gy. Systematic review and meta-analysis are the highest level
century. To optimise and enhance decision-making for medical
of evidence possible.
professionals, there is emphasis on evidence-based medicine.
Although medical professionals are always looking for the lat-
est research, they can feel overwhelmed with the amount of
Steps in conducting a systematic review
and meta-analysis
* Y. Bashir
[email protected] Research question

1
Professorial Surgical Unit, Department of Surgery, Tallaght Hospital, The first step in a systematic review and meta-analysis is to
The University of Dublin trinity College, Dublin, Ireland formulate a research question. Although it may appear simple,
448 Ir J Med Sci (2018) 187:447–452

Meta-Analysis include any restriction based on geography, language,

Systemac
Review age, study design (RCT’s only or other study designs as
RCT
well), only human studies and any other criteria you will
MulcentreStudies apply. Clarity about the inclusion and exclusion criteria
Observaonal Studies will help minimise bias.
Uncontrolled Trials
Non randomised trials
Focus group Studies Search strategy
Descripve Studies

Case Studies This step is the backbone of any systematic review and meta-
Expert Opinion
Studies of Poor Methodological Quality analysis. The main aim of all this exercise is to have an ex-
haustive and reproducible search. You need to make a balance
Fig. 1 Hierarchy of evidence showing the ranking in research
methodology between sensitivity and specificity. A narrow very specific
search may not include important articles on topic, which
it is the most vital element in the whole project. Data should might not have been defined correctly in the mesh with their
summarise the available literature on the topic. This gives you keywords. On the other hand, if your search is too sensitive,
an opportunity to gather the research available and conduct a you might have thousands of articles in your hand at the end of
qualitative analysis and perform statistical tests and finally search. And the shear burden of going through them might put
come up with a quantitative result about the question under you off the meta-analysis. At this point, do not hesitate to get
consideration. help from an experienced librarian. You have huge number of
databases; PubMed, Ovid Medline, EMBASE, Web of
Development of protocol Science, SCOPUS and Cochrane Library are only few to
name. Your librarian can help you in developing search strat-
The PICOS structure has been developed for formulating the egy with the help of keywords (BMeSH^ terms) related to
search question. PICOS model is a search structuring tool, each component of PICOS. On the other hand, if that is not
which helps you in making a purposeful and useful search the case in the database you are using, all the alternative words
possible. In this PICOS model, P stands for population under should be included in the search to avoid missing important
consideration, I for intervention, C for comparison group, O articles. Although most part of the search is done on electronic
for outcome and S for study design. It also includes making databases, still you need to do hand search through the bibli-
inclusion and exclusion criteria. This will help in minimising ography of the relevant articles to make sure you include all
bias in the systematic review and meta-analysis. You should the relevant articles. The search strategy for at least one of the
decide to use one of the standardised reporting systems. The databases should be included at the end of the meta-analysis as
most commonly used is PRISMA [3] protocol. an appendix to maintain transparency and reproducibility.
Figure 2 shows a search strategy for a meta-analysis in
Cochrane library [5]. Patients with venous ulcers given patient
Registration of the systematic review and meta-analysis
education were compared with patients without that and com-
pliance was analysed. Search strategies are added to the ap-
Always be smart: do not let anyone duplicate your study.
pendices of the articles to make them transparent and
PROSPERO [4] is an international prospective register of sys-
reproducible.
tematic reviews. This register makes sure that systematic re-
view topics are not duplicated. So, once you have finalised
your topic, the next step should be to register it. That will Screening
identify any similar systematic review and will avoid you
duplicating a topic which is an ongoing project for another Once the titles and abstracts of all articles are retrieved by
researcher. When you are writing, or reporting your systematic the search strategy, the screening process starts. It should
review and meta-analysis, you need to mention the be done by two researchers to minimise bias. Screening
PROSPERO registration number. titles and abstracts of all articles should be carried out and
all irrelevant articles should be removed. If there is any
Inclusion and exclusion criteria disagreement about any study, the decision should be
made after discussion and with consensus between two
As discussed earlier, you must develop a protocol for your researchers or by a third researcher. The full text of the
systematic review and meta-analysis. This protocol will articles selected after the initial screening be retrieved. A
define the outcome of interest. At this point, you should librarian’s assistance will be helpful in using the library’s
define the inclusion and exclusion criteria. This will document supply service.
Ir J Med Sci (2018) 187:447–452 449

Fig. 2 Search strategy for a

Cochrane study explaining all the
strategy used in searching of
databases

Final selection Analysing the data

At this stage, the full text article of the included papers must be Analysing the data can be a difficult experience for many med-
read carefully to see if they meet your inclusion criteria. The ical professionals. It involves either using Hunter and Schmidt
reason for exclusion of any article should be explained. All the [6] or Hedges and colleagues [7] using random effect or fixed
articles meeting inclusion criteria should be taken for qualita- effect models. This involves a lot of statistical work and use of
tive analysis (systematic review) and later if providing enough some advance mathematical formulas, which is usually beyond
information quantitative analysis (meta-analysis). the scope of a medical professional. In this era of computer
technology, things have been simplified. There are multiple
Data extraction software packages available that can help in extracting results.
RevMan, Meta-Analyst, Mix, Stata/Win BUGS and
Data extraction is a vital part of meta-analysis. A data extrac- Comprehensive Meta-Analysis are a few of them [8]. There
tion form should be designed and agreed on by all the authors. are cost differences between these packages, so the decision
Data extraction must be performed independently by two re- on which to use depends on available funding. Some of them
searchers. The first element of the data extraction form in- like RevMan and Open Meta-Analyst are free for use. If you
cludes information about the paper (journal of publication, want to use them, you might need tutorials or a short course.
year of publication, country of research, etc.). The latter part You also need to decide what effect model you are going to
should include information regarding the outcomes of interest. use. There are two available effect models: the fixed effect
model and the random effect model. If sample size is large
Quality assessment and there is very small or negligible inter-study heterogeneity,
you can use fixed model effect. Otherwise, you should use
All the studies that are included in the meta-analysis should be random effect size.
assessed for the quality. There are multiple different tools avail-
able that can help researchers in forming objective assessment Forest plots
of the study. The Newcastle-Ottawa Scale, Jadad Scale and
PEDro scale are a few of the most established scoring systems. A forest plot is a method of graphically representing the effects of
They aim to standardise quality assessment of studies. an intervention. The analysing software can produce a forest plot
450 Ir J Med Sci (2018) 187:447–452

Fig. 3 Forest plot shows that studies favour CNV over PPH but as the diamond crosses the midline, result is inconclusive

for you with a few clicks at the end of the analysis. Historically, not. It can be measured either as an odds ratio or risk ratio. The
horizontal lines and boxes were made for individual studies and odds ratio describes the odds of benefitting from the procedure
for the overall effect in the meta-analyses [9]. But the problem or intervention in question, and the risk ratio provides relative
with this way of representing the result is that the smaller studies risk of failure.
with larger confidence interval were depicted as big box on the These ratios can be only used if the data is dichotomous. If
result and attract more attention, while larger studies with smaller the data is continuous, then you need to use mean difference in
confidence intervals have smaller boxes. This problem was mean of the values. Whether your result favours the hypothe-
solved by changing to the forest plot. The forest plot concept sis or not can be easily seen by where the Bdiamond^ plots in
was proposed by Stephan Evans at Royal Statistical Society your forest plot are. The peak of your diamond represents the
Medical Section 1983 Meeting in London. It was based on a cumulative effect size, while the sides which represent the
modified box plot idea [10]. 95% confidence interval are also shown. If your diamond is
towards favour hypothesis but it crosses the midline, it be-
Interpretation of the forest plot comes equivocal and non-conclusive. As can be seen in Fig.
3, the diamond crosses the midline, so the result is not con-
Once the data is analysed, you need to interpret and know clusive. Although the peak of the diamond is on right side that
what the important features in the results are. This will direct favours CNV [11], the 95% confidence interval crosses the
the presentation of the results in a more attractive and inter- midline, which indicates that the CNV is not always an effec-
esting manner. Firstly, does the result favour the hypothesis or tive intervention.

Fig. 4 Forest plot on the right side clearly shows that results from studies favour pCR over no pCR
Ir J Med Sci (2018) 187:447–452 451

Fig. 5 In this forest plot, the

diamond of result is on the left
side favouring radial artery over
femoral artery

If the entire diamond is on the favoured side, it means Funnel plot

that the 95% confidence interval is fully to the right of the
line of no effect on that side, and only then is there a A funnel plot is a simple scatter plot of the intervention effect
positive result. It can be seen in Fig. 4 that the whole estimates from individual studies against some measure of
diamond is on the right side which favours pCR [12] each study’s size or precision [16]. It is a measure to see the
and all of 95% confidence interval is on the right of the degree of bias in the studies included in the meta-analysis. If
line of no effect. there is minimal bias and studies are quite similar, the scatter
In another example in Fig. 5, you can see that the complete plot should look like an inverted funnel. As can be seen in Fig.
diamond along with 95% confidence interval is to the left of 6, the studies are evenly scattered, forming a shape of an
the line favouring radial artery over ulnar artery [13]. inverted funnel. This indicates that there is a minimal bias in
The next value you need to know is heterogeneity. the studies involved in this meta-analysis [17].
Heterogeneity is defined as characteristic of being non- The larger-sized studies are placed towards the top of the
uniform and dissimilar. In a meta-analysis, it shows dissimi- funnel plot, while smaller studies with minimal effect are
larity among the studies involved in the meta-analysis. I2 is the placed towards the base. If there is bias because of smaller
value which determines heterogeneity among the studies in- studies showing no statistically significant effects, then such
cluded in the meta-analysis. I2 is defined as the percentage of publication bias will lead to an asymmetrical appearance of
variation across studies that is due to heterogeneity rather than the funnel plot. As can be seen in Fig. 7, there is asymmetrical
chance [14, 15]. I2 is an intuitive and simple expression of the plotting of the studies in the funnel plot, indicating bias [18].
inconsistency of the study results.

Additional statistical analysis

Additional statistical analysis of the data pooled from all the

studies can be done using SPSS. If the data available is

Fig. 6 Evenly spaced and scattered studies in funnel plot showing Fig. 7 Asymmetrical plotting of studies in funnel plot indicating bias in
minimum bias in the result of meta-analysis result of meta-analysis
452 Ir J Med Sci (2018) 187:447–452

categorical (nominal), the chi-square test is the preferred test 2. Evans D (2003) Hierarchy of evidence: a framework for ranking evi-
dence evaluating healthcare interventions. J Clin Nurs 12(1):77–84
but sometimes the Fisher Exact must be used if assumption is
3. Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew
violated. Alternatively, other test of significance and indepen- M et al (2015) Preferred reporting items for systematic review and
dence like independent sample t test, paired sample t test and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Rev
etc. can be done where appropriate. 4(1):1
4. Booth A, Clarke M, Dooley G, Ghersi D, Moher D, Petticrew M
et al (2012) The nuts and bolts of PROSPERO: an international
Reference management software prospective register of systematic reviews. Syst Rev 1(1):2
5. Weller CD, Buchbinder R, Johnston RV (2013) Interventions for
Other useful software that can be very beneficial for your helping people adhere to compression treatments for venous leg
ulceration. Cochrane Database Syst Rev. doi:10.1002/14651858.
work is reference management software. There are multiple
CD008378.pub2
options available including EndNote, Mendeley, ReadCube, 6. Schmidt FL, Le H (2004) Software for the Hunter-Schmidt meta-
Reference Manager and Bookends. They help you in manag- analysis methods. University of Iowa, Department of Management
ing the references in an effective way. And, there is a require- & Organization, Iowa City, IA 42242
7. Hedges LV (1984) Advances in statistical methods for meta-analy-
ment of different referencing style by the journals. These al-
sis. N Dir Eval 1984(24):25–42
low you to change between styles seamlessly in case you 8. Wallace BC, Schmid CH, Lau J, Trikalinos TA (2009) Meta-
choose a different journal for publication. Analyst: software for meta-analysis of binary, continuous and diag-
nostic data. BMC Med Res Methodol 9(1):80
9. Lewis J, Ellis S (1982) A statistical appraisal of post-infarction beta-
Write it up
blocker trials. Prim Cardiol (suppl 1):31–37
10. McGill R, Tukey JW, Larsen WA (1978) Variations of box plots.
Once you have done all the above-mentioned analysis, act Am Stat 32(1):12–16
quickly and write it up. Provide information about the search 11. Shao WJ, Li GC, Zhang ZK, Yang BL, Sun GD, Chen YQ (2008)
Systematic review and meta-analysis of randomized controlled tri-
strategy used and include it in an appendix. Inclusion and
als comparing stapled haemorrhoidopexy with conventional
exclusion criteria must be very clearly reported. Use tables haemorrhoidectomy. Br J Surg 95(2):147–160
and graphical representations to show your results. Your result 12. Martin S, Heneghan H, Winter D (2012) Systematic review and
should include the inclusion and 95% confidence interval and meta-analysis of outcomes following pathological complete re-
sponse to neoadjuvant chemoradiotherapy for rectal cancer. Br J
at the end, a concise but comprehensive conclusion should be
Surg 99(7):918–928
made based on the results. Use the PRISMA reporting check- 13. Jolly SS, Amlani S, Hamon M, Yusuf S, Mehta SR (2009) Radial
list as a guideline in writing it up. versus femoral access for coronary angiography or intervention and
the impact on major bleeding and ischemic events: a systematic
Acknowledgments The authors also wish to acknowledge Ms. Jean review and meta-analysis of randomized trials. Am Heart J
McMahon of Tallaght Hospital Library for her help in proofreading of 157(1):132–140
the document. 14. Higgins J, Thompson SG (2002) Quantifying heterogeneity in a
meta-analysis. Stat Med 21(11):1539–1558
15. Higgins J, Thompson SG, Deeks JJ, Altman DG (2003) Measuring
Compliance with ethical standards
inconsistency in meta-analyses [journal article as teaching resource,
deposited by John Flynn]. Br Med J 327:557–560
Funding sources No funding 16. Sterne JA, Harbord RM (2004) Funnel plots in meta-analysis. Stata
J 4:127–141
17. Sterne JA, Egger M (2001) Funnel plots for detecting bias in meta-
analysis: guidelines on choice of axis. J Clin Epidemiol 54(10):
References 1046–1055
18. Sterne JA, Gavaghan D, Egger M (2000) Publication and related
1. Field AP, Gillett R (2010) How to do a meta-analysis. Br J Math bias in meta-analysis: power of statistical tests and prevalence in the
Stat Psychol 63(3):665–694 literature. J Clin Epidemiol 53(11):1119–1129