BMC Medical Research

Methodology

BioMed Central

Open Access

Software

Meta-DiSc: a software for meta-analysis of test accuracy data

Javier Zamora*1, Victor Abraira1, Alfonso Muriel1, Khalid Khan2 and
Arri Coomarasamy2
Address: 1Clinical Biostatistics Unit, Ramn y Cajal Hospital, Madrid, Ctra. Colmenar km 9.100 Madrid 28034, Spain and 2University of
Birmingham and Birmingham Women's Hospital, Edgbaston, Birmingham, UK
Email: Javier Zamora* - [email protected]; Victor Abraira - [email protected]; Alfonso Muriel - [email protected];
Khalid Khan - [email protected]; Arri Coomarasamy - [email protected]
* Corresponding author

Published: 12 July 2006

BMC Medical Research Methodology 2006, 6:31

doi:10.1186/1471-2288-6-31

Received: 31 March 2006

Accepted: 12 July 2006

This article is available from: http://www.biomedcentral.com/1471-2288/6/31

2006 Zamora et al; licensee BioMed Central Ltd.
This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0),
which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract
Background: Systematic reviews and meta-analyses of test accuracy studies are increasingly being
recognised as central in guiding clinical practice. However, there is currently no dedicated and
comprehensive software for meta-analysis of diagnostic data. In this article, we present Meta-DiSc,
a Windows-based, user-friendly, freely available (for academic use) software that we have
developed, piloted, and validated to perform diagnostic meta-analysis.
Results: Meta-DiSc a) allows exploration of heterogeneity, with a variety of statistics including chisquare, I-squared and Spearman correlation tests, b) implements meta-regression techniques to
explore the relationships between study characteristics and accuracy estimates, c) performs
statistical pooling of sensitivities, specificities, likelihood ratios and diagnostic odds ratios using fixed
and random effects models, both overall and in subgroups and d) produces high quality figures,
including forest plots and summary receiver operating characteristic curves that can be exported
for use in manuscripts for publication. All computational algorithms have been validated through
comparison with different statistical tools and published meta-analyses. Meta-DiSc has a Graphical
User Interface with roll-down menus, dialog boxes, and online help facilities.
Conclusion: Meta-DiSc is a comprehensive and dedicated test accuracy meta-analysis software. It
has already been used and cited in several meta-analyses published in high-ranking journals. The
software is publicly available at http://www.hrc.es/investigacion/metadisc_en.htm.

Background
Accurate diagnosis forms the basis of good clinical care, as
without it one can neither prognosticate correctly nor
choose the right treatment. Indeed, a wrong diagnosis can
harm patients by exposing them to inappropriate or suboptimal therapy [1]. Thus studies of diagnostic accuracy,
and particularly their systematic reviews and meta-analyses, are being recognised as instrumental in underpinning
evidence-based clinical practice. Initiatives such as STARD

[2] and developments within the Cochrane Collaboration

[3] to accept protocols and reviews of test accuracy studies
highlight the emphasis being given to evidence-based
diagnosis.
Currently, there is only one test accuracy meta-analysis
package, Meta-Test [4], which addresses some of the
unique statistical issues related to test accuracy, such as
pooling of sensitivities and specificities and summary
Page 1 of 12
(page number not for citation purposes)

BMC Medical Research Methodology 2006, 6:31

receiver operating characteristics (sROC) analysis. However, it is a DOS-based application with an interface that
many find difficult to use, and integrate into Windowsbased applications. Moreover, it lacks crucial analytical
tools such as pooling of likelihood ratios (LRs), tests for
heterogeneity and meta-regression facilities.
We, therefore, developed, piloted and validated a comprehensive, Windows-based test accuracy meta-analysis software, Meta-DiSc, which is presented in this article, with a
worked example.

Implementation
Meta-DiSc software was created in Microsoft Visual Basic
6, and some mathematical routines have been linked
from the NAG C mathematical library [5]. The software is
distributed as a single file, downloadable freely from URL:
Its
http://www.hrc.es/investigacion/metadisc_en.htm.
installation is simple, guided by onscreen instructions.
The programme has a user-friendly interface with rolldown menus, dialog boxes and online HTML compiled
help files. These help files include a user manual and a
description of the implemented statistical methods.
Meta-DiSc allows data entry into its datasheet in three different ways: a) directly by typing data into the datasheet
using the keyboard, b) copying from another spreadsheet
(e.g. Microsoft Excel) and pasting into Meta-DiSc datasheet, or c) importing text files from other sources (for example, in the comma delimited format). Several variables
can be defined in the datasheet, including study identifiers, accuracy data from each study (true positives, false
positives, true negatives and false negatives) and study
level co-variates, such as those defining population spectrum or methodological quality of the studies.
Once the data have been entered into the datasheet of
Meta-DiSc, various statistical analyses can be implemented (Figure 1). The implementation of these statistical
procedures needs to be carefully thought through and
judicious, as it may be inappropriate (or indeed misleading) to use all the procedures (particularly statistical pooling) in all reviews. Meta-DiSc provides analysts with
adequate tools to assess the appropriateness of pooling.
Readers interested in details of these methods are referred
to statistical methods section of the help files (also available as a PDF standalone document [6] and to existing
texts and guidelines on diagnostic meta-analysis [7-10].
Describing the results of individual studies
When describing accuracy results from several studies, it is
important to get an indication of the magnitude and precision of the accuracy estimates derived from each study,
as well as to assess the presence or absence of inconsistencies in accuracy estimates across studies (heterogeneity).

http://www.biomedcentral.com/1471-2288/6/31

Describing primary results and exploring heterogeneity:

Tabular results.
Forest plots (sensitivity, specificity, LRs, dOR).
ROC plane scatter-plots.
Cochran-Q, Chi-Square, Inconsistency index.
Filtering/subgrouping capacities.
Exploring Threshold effect:
Spearman correlation coefficient
ROC plane plots
sROC curve fitting. Area under the curve (AUC) and Q*.
Meta-regression analysis:
Univariate and multivariate Moses & Littenberg model
(weighted or unweighted).
Statistical pooling of indices:
Fixed effect model.
Random effect model.

Figure 1 tools in Meta-DiSc

Available
Available tools in Meta-DiSc. Tools implemented in the
software Meta-DiSc to perform different steps of meta-analysis of diagnostic tests accuracy.

As accuracy estimates are paired and often inter-related

(sensitivity and specificity, or LR positive and LR negative), it is necessary to report these simultaneously [11].
One accuracy measure that combines these paired measures is diagnostic odd ratio (dOR) [12], which has limited
clinical use, although useful in procedures like metaregression (see below).
Meta-DiSc computes accuracy estimates and confidence
intervals from individual studies and shows results either
as numerical tabulations or graphical plots in two formats: a) forest plots, for sensitivities, specificities, LRs or
dOR, with respective confidence intervals; and b) plots of
individual study results in ROC space, with or without an
sROC curve.
Exploring heterogeneity (threshold effect)
Exploring heterogeneity is a critical issue to a) understand
the possible factors that influence accuracy estimates, and
b) to evaluate the appropriateness of statistical pooling of
accuracy estimates from various studies. One of the primary causes of heterogeneity in test accuracy studies is
threshold effect, which arises when differences in sensitivities and specificities or LRs occur due to different cut-offs
or thresholds used in different studies to define a positive
(or negative) test result. When threshold effect exists,
there is a negative correlation between sensitivities and
specificities (or a positive correlation between sensitivities
and 1-specificities), which results in a typical pattern of
"shoulder arm" plot in a sROC space [8]. It is worth noting that correlation between sensitivity and specificity
could arise due to a number of reasons other than threshold (e.g. partial verification bias, different spectrum of
patients or different settings).

Page 2 of 12
(page number not for citation purposes)

BMC Medical Research Methodology 2006, 6:31

Meta-DiSc allows assessment for threshold effect in three

different ways: a) visual inspection of relationship
between pairs of accuracy estimates in forest plots. If
threshold effect is present, the forest plots will show
increasing sensitivities with decreasing specificities, or
vice versa. The same inverse relationship will be apparent
with LR positive and LR negative; b) representation of
accuracy estimates from each study in a sROC space a
typical "shoulder arm" pattern would suggest presence of
threshold effect; and c) computation of Spearman correlation coefficient between the logit of sensitivity and logit of
1-specificity. A strong positive correlation would suggest
threshold effect.
Exploring for heterogeneity (other than threshold effect)
Apart from variations due to threshold effect, there are
several other factors that can result in variations in accuracy estimates amongst different test accuracy studies in a
review. These reasons include chance as well as variations
in study population (e.g. severity of disease and co-morbidities), index test (differences in technology, assays,
operator etc.), reference standard, and the way a study was
designed and conducted [13]. Since such heterogeneity is
almost always present in accuracy systematic reviews, testing for the presence and the extent of heterogeneity of
results between primary studies, prior to undertaking any
meta-analysis, is a critical part of any diagnostic review, as
is exploration of the possible causes of heterogeneity [14].

Meta-DiSc allows users to test for heterogeneity amongst

various studies in two different ways: a) Visual inspection
of forest plots of accuracy estimates. If the studies are reasonably homogeneous, the accuracy estimates from individual studies will lie along a line corresponding to the
pooled accuracy estimate. Large deviations from this line
will indicate possible heterogeneity; b) statistical tests,
including Chi-square and Cochran-Q, which are automatically implemented during analysis to evaluate if the differences across the studies are greater than expected by
chance alone. A low p-value will suggest presence of heterogeneity beyond what could be expected by chance
alone. In addition to these heterogeneity statistics, MetaDiSc computes the inconsistency index (I-squared) which
has been proposed as a measure to quantify the amount
of heterogeneity [15].
Meta-regression
If substantial heterogeneity is found to be present from
the analyses detailed above, then reasons for such heterogeneity can be explored by relating study level co-variates
(e.g., population, test, reference standard or methodological features) to an accuracy measure, using meta-regression techniques. The accuracy measure that is normally
used is dOR, as it is a unitary measure of diagnostic performance that encompasses both sensitivity and specifi-

http://www.biomedcentral.com/1471-2288/6/31

city or both LR positive and LR negative. Using dOR as a

global measure of accuracy is a suitable method to compare the overall diagnostic accuracy of different tests [13].
However, its use is limited because it cannot be used
directly in clinical practice and, furthermore, possible
opposing effects of a study characteristic on sensitivity or
specificity may be masked by using dOR.
Meta-DiSc implements meta-regression using a generalization of Littenberg and Moses Linear model [8,13]
weighted by inverse of the variance or study size or
unweighted. Random effects between studies can be estimated by different methods and added to the weighting
scheme [16]. Estimations of coefficients of the model are
performed by least squares method as implemented in
NAG mathematical routines. The outcome variable is
ln(dOR) which is related via a linear model to any
number of study level covariates, and optionally including the variable representing threshold effect [13]. The
outputs from meta-regression modelling in Meta-DiSc are
the co-efficients of the model, as well as ratio of dOR
(rdOR) with respective confidence intervals. If a particular
study level co-variate is significantly associated with diagnostic accuracy, then its co-efficient will have a low pvalue, and the rdOR will give a measure of magnitude of
the association.
More advanced meta-regression techniques such as Hierarchical sROC model [17] and bivariate analysis of sensitivity and specificity [18] has been developed. These
methods overcome some of the statistical shortcomings
inherent to Littenberg and Moses model [8,19].
Statistical pooling
Statistical pooling is not always appropriate or necessary
in every systematic review of test accuracy studies. However, when used appropriately, pooling can provide useful
summary information. The necessary precondition for
simple pooling (weighted averaging) of each of sensitivities, specificities, LR positives and LR negatives, is that the
studies and results are reasonably homogeneous (i.e. no
substantial heterogeneity, including threshold effect, is
present). If heterogeneity due to threshold effect were
present, the accuracy data can be pooled by fitting a sROC
curve and summarising that curve by means of the Area
Under the Curve (AUC) or using other statistics such as
the Q* index [19] (i.e. the point of the curve in which sensitivity equals specificity). If there is heterogeneity due to
sources other than threshold effect, then pooling should
only be attempted within homogeneous subsets, which
would normally have been defined a priori.

Meta-DiSc has comprehensive functionality for statistical

pooling: a) It allows pooling of sensitivities, specificities,
LR positive and LR negative each separately, using either

Page 3 of 12
(page number not for citation purposes)

BMC Medical Research Methodology 2006, 6:31

http://www.biomedcentral.com/1471-2288/6/31

fixed or random effect [10,20] models. The output from

these analyses are presented numerically in tables, and
graphically as forest plots. Pooled estimates are provided
with their respective confidence intervals; b) It implements several ways to fit a sROC curve when threshold
effect is present. Default option is to compute a symmetrical sROC curve after fitting the linear model proposed by
Littenberg and Moses. However, users can choose different options to fit this curve, for example, combining individual dORs by the Mantel-Haenszel or the DerSimonian
Laird methods [10,20] to estimate an overall dOR, and
then fitting an sROC curve. When the dOR changes with
diagnostic threshold, the sROC curve is asymmetrical.
Meta-DiSc allows the user to check for asymmetry of the
sROC curve, and fit an asymmetrical sROC curve if appropriate. Finally, Meta-DiSc allows estimation of AUC and
the Q* index, along with their standard errors, as a summary measure of global accuracy which also aids inter-test
comparisons; c) Meta-DiSc allows pooling of various
summary measures within subgroups defined by study
level co-variates with the help of a filter utility.

Wherever possible, the results of the above statistical procedures were validated using different general purpose statistical software such as STATA (ver 8.2) and SAS (8.2)
using actually published and simulated data sets (Table
1).

Results
We illustrate the various procedures that Meta-DiSc
implements in a case-study of ultrasound test in the diagnosis of uterine pathology [21,22]. Ultrasound measurement of the lining of the uterus (endometrium) can
predict pathology such as endometrial hyperplasia (a precancerous condition) or cancer. The greater the thickness
of endometrium, the more likely that the target condition
is present. Various thresholds (such as 3, 4 or 5 mm etc)
have been used to define a positive ultrasound result.
A systematic review of test accuracy studies identified 57
studies. Figure 2 shows a datasheet in Meta-DiSc which
has been loaded with information from these 57 studies.
The information includes study identifiers, accuracy data,

Table 1: Validation of statistical procedures. Validation of different statistical procedures using a simulated data-set. Results of MetaDiSc (version 1.4) are compared with those obtained with metan (version 1.86) and metareg (version 1.06) STATA commands. Prior
to the analyses, all four cells of all studies were added with 1/2 to avoid division by zero when computing some indices or standard
errors. Meta-DiSc and STATA data-set are provided as additional files [see Additional file 1] and [see Additional file 2].

Results
Procedure

Meta-DiSc (version 1.4)

STATA (ver 8.2)

Pooled +ve LR
(95%(CI)
Tau-square
Cochrane-Q

2.447
(2.085 2.871)
0.0932
139.71

2.447
(2.085 2.871)
0.0932
139.71

Pooled -ve LR
(95%(CI)
Tau-square
Cochrane-Q

0.157
(0.095 0.257)
0.4631
33.00

0.157
(0.095 0.257)
0.46357
33.07

Pooled +ve LR
(95%(CI)
Cochrane-Q

2.330
(2.208 2.459)
139.71

2.330
(2.208 2.459)
139.71

Pooled -ve LR
(95%(CI)
Cochrane-Q

0.105
(0.073 0.149)
33.00

0.104
(0.073 0.148)
33.07

Tau-Square
Constant coefficient (SE)
S coefficient (SE)
Covariable coefficient (SE)

0.1141
2.520 (0.8370)
0.330 (0.1912)
-0.036 (0.0904)

0.1141
2.5197 (0.83699)
0.3304 (0.19123)
-0.0355 (0.09041)

Random Effect Model

Fixed Effect Model

Meta-Regression1

(1) Meta-regression was weighted by the inverse of the variance of dOR and between study variance was estimated by REML.

Page 4 of 12
(page number not for citation purposes)

BMC Medical Research Methodology 2006, 6:31

http://www.biomedcentral.com/1471-2288/6/31

Figure 2 datasheet
Meta-Disc
Meta-Disc datasheet. Meta-DiSc data set with details of test accuracy studies of ultrasound in the prediction of endometrial
cancer.

thresholds, and some study level co-variates (such as hormone replacement therapy use).
As the first step in the analysis, we have used Meta-DiSc to
present accuracy measures from each individual study in
forest plots for sensitivities (figure 3a), specificities (figure
3b), LRs (figures 4a and 4b) and dOR (figure 5). All these
indices can also be represented in tabular form as shown
in table 2. Although the forest plots and the tables contain
a pooled summary at the bottom, at this early stage in the
analysis, it is recommended that the plots are used to
obtain a general overview of the accuracy estimates from
each study, and the interpretation of the pooled summary
is left to later stages of analysis.
The next step is the representation of sensitivity against 1specificity from each study in a ROC space (figure 6),
which can be used for exploration for threshold effect. The

pattern of the points in this plot suggest a "shoulder-arm"

shape, indicating the possibility of threshold effect. We,
therefore, performed a Spearman rank correlation as a further test for threshold effect, and found that there was further indication of threshold effect (Table 3, Spearman
correlation coefficient = 0.394; p = 0.006). Having found
some clues about the presence of threshold effect, we now
focus on a subgroup of 21 studies that used a singular
threshold of >5 mm to define test positivity. Although an
explicit threshold of 5 mm was used in these studies, there
can still be an implicit threshold effect due to, for example, variation in the interpretation of the test results.
Therefore, within this subgroup with an explicit threshold
of 5 mm, it is still recommended that the above explorations for threshold effect are undertaken. We performed
such analyses for this subgroup in Meta-DiSc, and found
no evidence of further threshold effect (data not shown).
There are a number of other more advanced methods not

Page 5 of 12
(page number not for citation purposes)

BMC Medical Research Methodology 2006, 6:31

http://www.biomedcentral.com/1471-2288/6/31

A
Sensitivity (95% CI)
Auslender
Zannoni
Bakour
Botsis
Fistonic
Garuti
Granberg
Guner
Haller
Tsuda
Varner
Abu Ghazzeh
Briley
Cacciatore
DeSilva
Granberg
Grigoriou
Gu
Gupta
Hnggi
Ivanov
Karlsson
Loverro
Malinova
Merz
Nasri
Nasri
Pertl
Suchocki
Taviani
Weber
Wolman
Moreles
Rudigoz
Todorova
Gruboeck
Chan
Degenhardt
Dijkhuizen
Brolmann
Ceccini
Masearetti
Mortakis
Schramm
Smith
Osmers
Seelbach-Gbel
Altuncu et al.

0,2

0,4
0,6
Sensitivity

0,8

1,00
0,98
1,00
1,00
1,00
0,98
1,00
1,00
1,00
0,93
1,00
1,00
1,00
1,00
0,33
1,00
1,00
1,00
0,67
0,86
1,00
0,93
1,00
1,00
1,00
1,00
1,00
0,95
1,00
1,00
0,98
1,00
0,91
0,78
1,00
0,82
1,00
0,86
1,00
1,00
0,94
1,00
1,00
0,62
1,00
1,00
0,95
0,83

Specificity (95% CI)

(0,79 - 1,00)
(0,90 - 1,00)
(0,72 - 1,00)
(0,63 - 1,00)
(0,77 - 1,00)
(0,91 - 1,00)
(0,97 - 1,00)
(0,82 - 1,00)
(0,79 - 1,00)
(0,68 - 1,00)
(0,16 - 1,00)
(0,03 - 1,00)
(0,48 - 1,00)
(0,40 - 1,00)
(0,01 - 0,91)
(0,63 - 1,00)
(0,86 - 1,00)
(0,59 - 1,00)
(0,09 - 0,99)
(0,64 - 0,97)
(0,69 - 1,00)
(0,68 - 1,00)
(0,86 - 1,00)
(0,95 - 1,00)
(0,77 - 1,00)
(0,59 - 1,00)
(0,54 - 1,00)
(0,74 - 1,00)
(0,88 - 1,00)
(0,16 - 1,00)
(0,91 - 1,00)
(0,40 - 1,00)
(0,71 - 0,99)
(0,40 - 0,97)
(0,16 - 1,00)
(0,48 - 0,98)
(0,80 - 1,00)
(0,71 - 0,95)
(0,63 - 1,00)
(0,69 - 1,00)
(0,70 - 1,00)
(0,29 - 1,00)
(0,59 - 1,00)
(0,42 - 0,79)
(0,40 - 1,00)
(0,87 - 1,00)
(0,83 - 0,99)
(0,36 - 1,00)

Pooled Sensitivity = 0,96 (0,94 to 0,97)

Chi-square = 108,51; df = 47 (p = 0,0000)
1 Inconsistency (I-square) = 56,7 %

Auslender
Zannoni
Bakour
Botsis
Fistonic
Garuti
Granberg
Guner
Haller
Tsuda
Varner
Abu Ghazzeh
Briley
Cacciatore
DeSilva
Granberg
Grigoriou
Gu
Gupta
Hnggi
Ivanov
Karlsson
Loverro
Malinova
Merz
Nasri
Nasri
Pertl
Suchocki
Taviani
Weber
Wolman
Moreles
Rudigoz
Todorova
Gruboeck
Chan
Degenhardt
Dijkhuizen
Brolmann
Ceccini
Masearetti
Mortakis
Schramm
Smith
Osmers
Seelbach-Gbel
Altuncu et al.

0,2

0,4
0,6
Specificity

0,8

0,51
0,53
0,49
0,88
0,19
0,33
0,52
0,47
0,26
0,63
0,54
0,38
0,51
0,27
0,74
0,76
0,67
0,27
0,64
0,79
0,58
0,65
0,84
0,49
0,43
0,66
0,61
0,27
0,12
0,54
0,39
0,64
0,61
0,74
0,50
0,88
0,62
0,66
0,49
0,53
0,71
0,58
0,58
0,50
0,54
0,50
0,44
0,97

(0,42 - 0,61)
(0,49 - 0,57)
(0,38 - 0,60)
(0,80 - 0,93)
(0,12 - 0,29)
(0,28 - 0,38)
(0,49 - 0,55)
(0,39 - 0,55)
(0,16 - 0,39)
(0,55 - 0,71)
(0,25 - 0,81)
(0,28 - 0,49)
(0,43 - 0,58)
(0,14 - 0,43)
(0,60 - 0,86)
(0,70 - 0,82)
(0,60 - 0,73)
(0,11 - 0,50)
(0,52 - 0,75)
(0,67 - 0,87)
(0,46 - 0,69)
(0,54 - 0,75)
(0,74 - 0,91)
(0,38 - 0,60)
(0,28 - 0,59)
(0,52 - 0,78)
(0,50 - 0,72)
(0,19 - 0,35)
(0,06 - 0,20)
(0,37 - 0,70)
(0,29 - 0,50)
(0,49 - 0,77)
(0,53 - 0,68)
(0,59 - 0,86)
(0,16 - 0,84)
(0,80 - 0,94)
(0,47 - 0,75)
(0,55 - 0,75)
(0,36 - 0,62)
(0,39 - 0,66)
(0,66 - 0,76)
(0,33 - 0,80)
(0,45 - 0,69)
(0,42 - 0,58)
(0,37 - 0,69)
(0,43 - 0,57)
(0,36 - 0,51)
(0,85 - 1,00)

Pooled Specificity = 0,54 (0,53 to 0,55)

Chi-square = 646,82; df = 47 (p = 0,0000)
1 Inconsistency (I-square) = 92,7 %

Figureplot
Forest
3
Forest plot. Forrest plot of sensitivities (3a) and specificities (3b) from test accuracy studies of ultrasound in the prediction of
endometrial cancer.

implemented in Meta-DiSc that allow to incorporate

explicitly information about tests thresholds defined
between or within studies [17].
As the next step, heterogeneity arising from factors other
than threshold effect is explored. We performed a visual
exploration of the forest plots of accuracy measures for
these 21 studies as well as statistical tests for heterogeneity
(Meta-DiSc output not shown). In addition, possible
sources of heterogeneity across the studies were explored
using meta-regression analysis with the following co-variates as predictor variables: use or non-use of hormone
replacement therapy (HRT); technique of ultrasound
measurement (single or double layer); and population
enrolment (consecutive or other). Results are shown in
Table 4, which suggest that the number of layers is
strongly associated with accuracy. The double layer technique is associated with two times higher accuracy com-

pared to single layer measurement (rdOR = 2.04; 95% CI:

1.014.13; p = 0.048)
The final step in the analysis is pooling if this is considered appropriate. We illustrate pooling of the LRs for negative test results in one homogenous subgroup of studies
of non-HRT users, with a test threshold of 5 mm, and
using a single layer technique (Figure 7). Finally, we demonstrate sROC curve fitting in the presence of threshold
effect for the whole data-set in Figure 8.

Discussion and conclusion

Meta-DiSc allows description of individual study results;
exploration of heterogeneity with a variety of statistics
including chi-square, I-squared and Spearman correlation
tests; implements meta-regression techniques to explore
the relationships between study characteristics and accuracy estimates; performs statistical pooling of sensitivities,

Page 6 of 12
(page number not for citation purposes)

BMC Medical Research Methodology 2006, 6:31

http://www.biomedcentral.com/1471-2288/6/31

A
Positive LR (95% CI)
Auslender
Zannoni
Bakour
Botsis
Fistonic
Garuti
Granberg
Guner
Haller
Tsuda
Varner
Abu Ghazzeh
Briley
Cacciatore
DeSilva
Granberg
Grigoriou
Gu
Gupta
Hnggi
Ivanov
Karlsson
Loverro
Malinova
Merz
Nasri
Nasri
Pertl
Suchocki
Taviani
Weber
Wolman
Moreles
Rudigoz
Todorova
Gruboeck
Chan
Degenhardt
Dijkhuizen
Brolmann
Ceccini
Masearetti
Mortakis
Schramm
Smith
Osmers
Seelbach-Gbel
Altuncu et al.

0,01

1
Positive LR

1,99
2,09
1,89
7,36
1,20
1,47
2,07
1,83
1,32
2,52
1,79
1,21
1,85
1,24
1,31
3,94
2,95
1,31
1,85
4,00
2,27
2,65
5,96
1,96
1,70
2,74
2,40
1,29
1,12
1,80
1,62
2,48
2,31
2,98
1,67
7,04
2,54
2,52
1,86
2,02
3,27
2,06
2,21
1,24
1,94
1,96
1,68
29,17

Negative LR (95% CI)

(1,62 - 2,45)
(1,92 - 2,28)
(1,49 - 2,41)
(4,44 - 12,21)
(1,04 - 1,38)
(1,36 - 1,59)
(1,93 - 2,21)
(1,57 - 2,14)
(1,12 - 1,56)
(1,96 - 3,22)
(0,84 - 3,83)
(0,54 - 2,75)
(1,40 - 2,47)
(0,88 - 1,75)
(0,24 - 6,96)
(2,93 - 5,28)
(2,43 - 3,57)
(0,96 - 1,79)
(0,78 - 4,35)
(2,47 - 6,47)
(1,69 - 3,05)
(1,94 - 3,63)
(3,65 - 9,73)
(1,59 - 2,42)
(1,29 - 2,24)
(1,83 - 4,10)
(1,71 - 3,37)
(1,11 - 1,50)
(1,03 - 1,22)
(0,98 - 3,30)
(1,37 - 1,90)
(1,56 - 3,96)
(1,85 - 2,90)
(1,64 - 5,43)
(0,73 - 3,81)
(3,69 - 13,42)
(1,78 - 3,64)
(1,86 - 3,41)
(1,39 - 2,49)
(1,49 - 2,74)
(2,65 - 4,02)
(1,10 - 3,87)
(1,60 - 3,06)
(0,90 - 1,71)
(1,25 - 3,00)
(1,70 - 2,27)
(1,45 - 1,94)
(4,09 - 208,02)

Random Effects Model

Pooled Positive LR = 2,09 (1,88 to 2,31)
Cochran-Q = 506,06; df = 47 (p = 0,0000)
100,0 Inconsistency (I-square) = 90,7 %
Tau-square = 0,0992

Auslender
Zannoni
Bakour
Botsis
Fistonic
Garuti
Granberg
Guner
Haller
Tsuda
Varner
Abu Ghazzeh
Briley
Cacciatore
DeSilva
Granberg
Grigoriou
Gu
Gupta
Hnggi
Ivanov
Karlsson
Loverro
Malinova
Merz
Nasri
Nasri
Pertl
Suchocki
Taviani
Weber
Wolman
Moreles
Rudigoz
Todorova
Gruboeck
Chan
Degenhardt
Dijkhuizen
Brolmann
Ceccini
Masearetti
Mortakis
Schramm
Smith
Osmers
Seelbach-Gbel
Altuncu et al.

0,01

1
Negative LR

0,06
0,03
0,08
0,06
0,17
0,05
0,01
0,05
0,11
0,11
0,31
0,65
0,16
0,37
0,90
0,07
0,03
0,22
0,52
0,18
0,08
0,10
0,02
0,01
0,08
0,10
0,12
0,20
0,14
0,31
0,04
0,16
0,15
0,30
0,33
0,21
0,04
0,21
0,11
0,09
0,09
0,22
0,11
0,76
0,19
0,04
0,12
0,17

(0,00 - 0,88)
(0,00 - 0,24)
(0,01 - 1,28)
(0,00 - 0,94)
(0,01 - 2,70)
(0,01 - 0,35)
(0,00 - 0,13)
(0,00 - 0,83)
(0,01 - 1,75)
(0,02 - 0,71)
(0,02 - 4,09)
(0,06 - 7,30)
(0,01 - 2,35)
(0,03 - 5,30)
(0,40 - 2,03)
(0,00 - 1,08)
(0,00 - 0,47)
(0,01 - 3,50)
(0,10 - 2,61)
(0,06 - 0,52)
(0,01 - 1,18)
(0,02 - 0,69)
(0,00 - 0,36)
(0,00 - 0,23)
(0,00 - 1,21)
(0,01 - 1,40)
(0,01 - 1,69)
(0,03 - 1,36)
(0,01 - 2,31)
(0,02 - 3,96)
(0,01 - 0,29)
(0,01 - 2,19)
(0,04 - 0,56)
(0,09 - 1,03)
(0,02 - 4,55)
(0,06 - 0,72)
(0,00 - 0,70)
(0,09 - 0,47)
(0,01 - 1,69)
(0,01 - 1,31)
(0,01 - 0,59)
(0,02 - 2,99)
(0,01 - 1,60)
(0,46 - 1,24)
(0,01 - 2,63)
(0,00 - 0,56)
(0,03 - 0,46)
(0,03 - 1,03)

Random Effects Model

Pooled Negative LR = 0,14 (0,09 to 0,21)
Cochran-Q = 112,11; df = 47 (p = 0,0000)
100,0 Inconsistency (I-square) = 58,1 %
Tau-square = 1,0710

Figureplot
Forest
4
Forest plot. Forrest plot of likelihood ratios for positive (4a) and negative (4b) test results from studies of ultrasound in the
prediction of endometrial cancer.

specificities, likelihood ratios and diagnostic odds ratios,

using fixed and random effects models, both overall and
in subgroups; and produces high quality figures, including forest plots and summary receiver operating characteristic curves that can be exported for use in manuscripts for
publication.
Meta-DiSc is an evolving software. As new diagnostic
meta-analytic methods become established over time,
they will be implemented into the program in the future.
For example, bivariate method of pooling sensitivity and
specificity [18] is currently being developed. We will carefully follow the progress in this field. Once accepted as an
established meta-analytic method, it will be implemented
in Meta-DiSc. On similar lines, methods of data extraction
from individual studies that only provide accuracy measures are currently being developed within our department. Once these methods have been verified, we will

implement this option to assist systematic reviewers in

extracting 2-by-2 tables from such studies.
Meta-DiSc is a comprehensive and dedicated test accuracy
meta-analysis software. All computational algorithms in it
have been validated through comparison with different
statistical tools and published meta-analyses. Its use and
citation in several meta-analyses published in high-ranking journals is evidence of external validation of its high
quality [23-28].

Availability and requirements

The software is publicly available at http://www.hrc.es/
investigacion/metadisc_en.htm.
Operating system: The software runs on Windows based
personal computers (Windows 95 or higher) with Pentium-class processor or equivalent, with minimum of 32

Page 7 of 12
(page number not for citation purposes)

BMC Medical Research Methodology 2006, 6:31

http://www.biomedcentral.com/1471-2288/6/31

Diagnostic OR (95% CI)

Auslender
Zannoni
Bakour
Botsis
Fistonic
Garuti
Granberg
Guner
Haller
Tsuda
Varner
Abu Ghazzeh
Briley
Cacciatore
DeSilva
Granberg
Grigoriou
Gu
Gupta
Hnggi
Ivanov
Karlsson
Loverro
Malinova
Merz
Nasri
Nasri
Pertl
Suchocki
Taviani
Weber
Wolman
Moreles
Rudigoz
Todorova
Gruboeck
Chan
Degenhardt
Dijkhuizen
Brolmann
Ceccini
Masearetti
Mortakis
Schramm
Smith
Osmers
Seelbach-Gbel
Altuncu et al.

0,01

1
Diagnostic Odds Ratio

34,78 (2,04 - 593,79)

62,15 (8,55 - 451,56)
22,47 (1,28 - 393,49)
115,48 (6,32 - 2.109,25)
7,00 (0,40 - 123,10)
29,25 (4,00 - 213,73)
246,16 (15,26 - 3.970,34)
34,36 (2,04 - 578,15)
11,91 (0,68 - 209,19)
23,75 (3,04 - 185,50)
5,77 (0,23 - 143,37)
1,86 (0,07 - 46,84)
11,26 (0,61 - 206,72)
3,39 (0,17 - 68,12)
1,46 (0,12 - 17,56)
53,86 (3,05 - 950,71)
98,32 (5,90 - 1.638,95)
5,91 (0,29 - 119,06)
3,54 (0,31 - 40,93)
22,00 (5,71 - 84,78)
29,00 (1,64 - 513,45)
25,74 (3,23 - 205,10)
258,78 (14,83 - 4.514,43)
135,80 (8,15 - 2.264,08)
21,90 (1,23 - 391,31)
28,85 (1,56 - 531,94)
20,60 (1,12 - 378,06)
6,56 (0,84 - 51,00)
7,96 (0,46 - 138,73)
5,81 (0,26 - 128,90)
39,29 (5,22 - 295,45)
15,81 (0,81 - 310,35)
15,43 (3,50 - 68,07)
9,92 (1,80 - 54,49)
5,00 (0,18 - 136,32)
34,20 (6,45 - 181,32)
56,54 (3,21 - 994,48)
12,22 (4,35 - 34,30)
16,46 (0,91 - 297,75)
23,38 (1,31 - 418,59)
37,28 (4,86 - 285,93)
9,47 (0,43 - 208,76)
20,41 (1,12 - 371,17)
1,64 (0,73 - 3,68)
10,38 (0,53 - 205,27)
55,00 (3,31 - 913,67)
14,26 (3,34 - 60,84)
170,00 (9,11 - 3.172,49)

Random Effects Model

Pooled Diagnostic Odds Ratio = 17,48 (11,59 to 26,37)
Cochran-Q = 70,56; df = 47 (p = 0,0147)
100,0 Inconsistency (I-square) = 33,4 %
Tau-square = 0,6179

Figure 5
Forrest
plot
Forrest plot. Forest plot of diagnostic odds ratios (dOR) from test accuracy studies of ultrasound in the prediction of
endometrial cancer.

Page 8 of 12
(page number not for citation purposes)

BMC Medical Research Methodology 2006, 6:31

http://www.biomedcentral.com/1471-2288/6/31

Table 2: Tabulation of Likelihood ratio for positive test result (LR+) with respective 95% confidence intervals from all test accuracy
studies included in systematic review of ultrasound for prediction of endometrial cancer.

Study

LR+

[95% Conf. Iterval.]

Auslender
Zannoni
Bakour
Botsis
Fistonic
Garuti
Granberg
Guner
Haller
Tsuda
Varner
Abu Ghazzeh
Briley
Cacciatore
DeSilva
Granberg
Grigoriou
Gu
Gupta
Hnggi
Ivanov
Karlsson
Loverro
Malinova
Merz
Nasri
Nasri
Pertl
Suchocki
Taviani
Weber
Wolman
Moreles
Rudigoz
Todorova
Gruboeck
Chan
Degenhardt
Dijkhuizen
Brolmann
Ceccini
Masearetti
Mortakis
Schramm
Smith
Osmers
Seelbach-Gbel
Altuncu et al.

1,994
2,092
1,895
7,360
1,200
1,471
2,066
1,834
1,321
2,517
1,795
1,215
1,855
1,239
1,306
3,937
2,946
1,307
1,846
4,000
2,273
2,649
5,957
1,963
1,697
2,740
2,400
1,293
1,120
1,802
1,618
2,481
2,312
2,981
1,667
7,036
2,543
2,516
1,859
2,017
3,267
2,059
2,213
1,241
1,938
1,964
1,680
29,167

1,623
1,919
1,490
4,437
1,045
1,358
1,935
1,569
1,118
1,964
0,842
0,538
1,396
0,877
0,245
2,933
2,430
0,956
0,783
2,472
1,691
1,936
3,648
1,591
1,287
1,833
1,711
1,115
1,027
0,983
1,374
1,556
1,845
1,638
0,729
3,689
1,779
1,856
1,389
1,487
2,655
1,096
1,602
0,899
1,252
1,699
1,455
4,089

-2,449
-2,280
-2,408
-12,208
-1,378
-1,593
-2,206
-2,144
-1,561
-3,225
-3,826
-2,745
-2,465
-1,752
-6,957
-5,284
-3,572
-1,787
-4,350
-6,473
-3,054
-3,627
-9,729
-2,421
-2,236
-4,096
-3,367
-1,499
-1,222
-3,304
-1,904
-3,956
-2,896
-5,426
-3,808
-13,422
-3,635
-3,411
-2,489
-2,736
-4,021
-3,866
-3,058
-1,714
-3,001
-2,271
-1,940
-208,02

(REM) pooled LR+

2,087

1,881

-2,315

% Weight
2,54
2,77
2,45
1,69
2,69
2,78
2,79
2,65
2,63
2,43
1,13
1,03
2,33
2,15
0,34
2,30
2,57
2,25
0,96
1,76
2,30
2,24
1,73
2,53
2,35
1,98
2,17
2,67
2,77
1,44
2,64
1,80
2,49
1,46
1,01
1,35
2,12
2,27
2,31
2,27
2,54
1,38
2,22
2,22
1,88
2,68
2,68
0,25

Heterogeneity chi-squared = 506,06 (d.f.= 47) p = 0,000

Inconsistency (I-square) = 90,7%
No. studies = 48.
Filter OFF
Add 1/2 only zero cell studies

Page 9 of 12
(page number not for citation purposes)

BMC Medical Research Methodology 2006, 6:31

http://www.biomedcentral.com/1471-2288/6/31

Table 3: Results of Spearman rank correlation of sensitivity against (1 specificity) to assess the threshold effect in all test accuracy
studies included in systematic review of ultrasound for prediction of endometrial cancer.

Var.

Coeff.

Std. Error

p-value

A
b(1)

2.412
0.187

0.292
0.101

8.266
1.857

0.0000
0.0697

Spearman correlation coefficient: 0,394 p-value = 0,006 (Logit(TPR) vs Logit(FPR)

Moses' model (D = a + bS)
Unweighted regression
Tau-squared estimate = 0,3540
(Convergence is achieved after 2 iterations)
Restricted Maximum Likelihood estimation (REML)
No. studies = 48
Filter OFF

Table 4: Results of meta-regression analysis for predicting the presence or absence of endometrial carcinoma with variables: use or
non-use of hormone replacement therapy (HRT); technique of ultrasound measurement (single or double layer); and population
enrolment (consecutive or other).

Meta-Regression(Inverse Variance weights) (1)

Var.
Coeff.

p-value

RDOR

[95%CI]

Cte.
S
Layers
Consecutive
HRT

0,1571
0,0208
0,0610
0,7398
0,4152

------2,03
1,23
1,38

------(0,97;4,27)
(0,35;4,26)
(0,63;3,06)

Meta-Regression(Inverse Variance weights) (2)

Var.
Coeff.

p-value

RDOR

[95%CI]

Cte.
S
Layers
HRT

0,1565
0,0194
0,0424
0,4152

------2,09
1,38

------(1,03;4,27)
(0,63;3,02)

Meta-Regression(Inverse Variance weights) (3)

Var.
Coeff.

p-value

RDOR

[95%CI]

Cte.
S
Layers

0,0999
0,0166
0,0482

------2,04

------(1,01;4,13)

0,857
0,263
0,709
0,206
0,324

0,849
0,253
0,739
0,320

0,959
0,258
0,712

Page 10 of 12
(page number not for citation purposes)

BMC Medical Research Methodology 2006, 6:31

Sensitivity
1

http://www.biomedcentral.com/1471-2288/6/31

ROC space

Positive LR (95% CI)

0,9
Abu Ghazzeh
Briley
Cacciatore
Grigoriou
Gu
Gupta
Hnggi
Ivanov
Karlsson
Loverro
Malinova
Nasri
Suchocki
Taviani
Weber
Wolman

0,8

0,7

0,6

0,5

1,21
1,85
1,24
2,95
1,31
1,85
4,00
2,27
2,65
5,96
1,96
2,74
1,12
1,80
1,62
2,48

(0,54 - 2,75)
(1,40 - 2,47)
(0,88 - 1,75)
(2,43 - 3,57)
(0,96 - 1,79)
(0,78 - 4,35)
(2,47 - 6,47)
(1,69 - 3,05)
(1,94 - 3,63)
(3,65 - 9,73)
(1,59 - 2,42)
(1,83 - 4,10)
(1,03 - 1,22)
(0,98 - 3,30)
(1,37 - 1,90)
(1,56 - 3,96)

0,4

0,3
0,01

0,2

0,1

1
Positive LR

Random Effects Model

Pooled Positive LR = 2,08 (1,59 to 2,72)
Cochran-Q = 240,88; df = 15 (p = 0,0000)
100,0 Inconsistency (I-square) = 93,8 %
Tau-square = 0,2593

Negative LR (95% CI)

0,2

0,4

0,6

0,8

Abu Ghazzeh
Briley
Cacciatore
Grigoriou
Gu
Gupta
Hnggi
Ivanov
Karlsson
Loverro
Malinova
Nasri
Suchocki
Taviani
Weber
Wolman

1-specificity

Figure
ROC
Space
6
ROC Space. Representation of sensitivity against (1-specificity) in Receiver Operating Characteristics space for each
study of ultrasound in the prediction of endometrial cancer.

0,01

Sensitivity
1

1
Negative LR

0,65
0,16
0,37
0,03
0,22
0,52
0,18
0,08
0,10
0,02
0,01
0,10
0,14
0,31
0,04
0,16

(0,06 - 7,30)
(0,01 - 2,35)
(0,03 - 5,30)
(0,00 - 0,47)
(0,01 - 3,50)
(0,10 - 2,61)
(0,06 - 0,52)
(0,01 - 1,18)
(0,02 - 0,69)
(0,00 - 0,36)
(0,00 - 0,23)
(0,01 - 1,40)
(0,01 - 2,31)
(0,02 - 3,96)
(0,01 - 0,29)
(0,01 - 2,19)

Random Effects Model

Pooled Negative LR = 0,14 (0,08 to 0,25)
Cochran-Q = 15,95; df = 15 (p = 0,3853)
100,0 Inconsistency (I-square) = 6,0 %
Tau-square = 0,0794

Figure 7
Forrest
plot
Forrest plot. Forrest plots of Likelihood ratios for positive
(7a) and negative (7b) test results in one homogenous subgroup of studies of non-HRT users, with a test threshold of
5 mm, and using a single layer technique.

SROC Curve

Symmetric SROC
AUC = 0,8012
SE(AUC) = 0,0562
Q* = 0,7369
SE(Q*) = 0,0491

0,9

0,8

0,7

0,6

0,5

0,4

0,3

0,2

0,1

0,2

0,4

0,6

0,8

1-specificity

Figurecurve
sROC
8
sROC curve. Receiver operating characteristics curve for
all studies included in systematic review of ultrasound for
prediction of endometrial cancer.

Page 11 of 12
(page number not for citation purposes)

BMC Medical Research Methodology 2006, 6:31

MB of RAM and minimum of 20 MB of hard disk space.

SVGA color monitor; minimum 800 600 screen resolution and 256 colors.

http://www.biomedcentral.com/1471-2288/6/31

9.

10.

Licence: Freeware for academic use.

Competing interests

11.

The author(s) declare that they have no competing interests.

12.

Authors' contributions

13.

JZ conceived the idea. AM, VA and JZ developed the software. AC and KSK tested the software on a number of
reviews and gave suggestions for improvements. All
authors participated in preparing this manuscript.

14.

15.

Additional material
Additional File 1
Meta-Disc data set. This file contains simulated data. It is provided to help
users to validate statistical procedures shown in table 1.
Click here for file
[http://www.biomedcentral.com/content/supplementary/14712288-6-31-S1.dsc]

Additional File 2
STATA data set. This file contains simulated data. It is provided to help
users to validate statistical procedures shown in table 1.
Click here for file
[http://www.biomedcentral.com/content/supplementary/14712288-6-31-S2.dta]

Acknowledgements
This work has been partly funded by Spanish Health Ministry Grants no
PI02/0954, G03/090 and PI04/1055.

16.
17.
18.

19.
20.
21.

22.

23.

24.

References
1.

2.

3.
4.
5.
6.
7.
8.

Thomson R, McElroy H, Sudlow M: Guidelines on anticoagulant

treatment in atrial fibrillation in Great Britain: variation in
content and implications for treatment.
BMJ 1998,
316:509-513.
Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig
LM, Lijmer JG, Moher D, Rennie D, de Vet HC: Towards complete
and accurate reporting of studies of diagnostic accuracy: The
STARD Initiative. Radiology 2003, 226:24-28.
Collaboration C: Methods Groups Newsletter. http://www
cochrane
org/newslett/MGNews-2004
pdf
2006
[http://
www.cochrane.org/newslett/MGNews-2004.pdf].
Lau J: Meta-Test. Boston: New England Medical Center; 1997.
The NAG C Library, Mark 6. Oxford: Numerical Algorithms
Group; 2004.
Zamora J, Muriel A, Abraira V: Meta-DiSc Statistical Methods.
2006
[ftp://ftp.hrc.es/pub/programas/metadisc/
MetaDisc_StatisticalMethods.pdf].
Irwig L, Tosteson ANA, Gatsonis C, Lau J, Colditz G, Chalmers TC,
Mosteller F: Guidelines for Metaanalyses Evaluating Diagnostic-Tests. Annals of Internal Medicine 1994, 120:667-676.
Moses LE, Shapiro D, Littenberg B: Combining independent studies of a diagnostic test into a summary ROC curve: data-analytic approaches and some additional considerations. Stat
Med 1993, 12:1293-1316.

25.

26.

27.

28.

Deville WL, Buntinx F, Bouter LM, Montori VM, de Vet HC, van der
Windt DA, Bezemer PD: Conducting systematic reviews of
diagnostic studies: didactic guidelines. BMC Med Res Methodol
2002, 2:9.
Deeks JJ: Systematic reviews of evaluations of diagnostic and
screening tests studies. In Systematic reviews in health care: metaanalysis in context 2nd Edition edition. Edited by: Egger M, Davey SG
and Altman DG. BMJ Books; 2001.
Honest H, Khan KS: Reporting of measures of accuracy in systematic reviews of diagnostic literature. Bmc Health Services
Research 2002, 2:.
Glas AS, Lijmer JG, Prins MH, Bonsel GJ, Bossuyt PM: The diagnostic odds ratio: a single indicator of test performance. J Clin Epidemiol 2003, 56:1129-1135.
Lijmer JG, Bossuyt PM, Heisterkamp SH: Exploring sources of heterogeneity in systematic reviews of diagnostic tests. Stat Med
2002, 21:1525-1537.
Dinnes J, Deeks J, Kirby J, Roderick P: A methodological review
of how heterogeneity has been examined in systematic
reviews of diagnostic test accuracy. Health Technol Assess 2005,
9:1-113.
Higgins JP, Thompson SG: Quantifying heterogeneity in a metaanalysis. Stat Med 2002, 21:1539-1558.
Thompson SG, Sharp SJ: Explaining heterogeneity in meta-analysis: a comparison of methods. Stat Med 1999, 18:2693-2708.
Rutter CM, Gatsonis CA: A hierarchical regression approach to
meta-analysis of diagnostic test accuracy evaluations. Stat
Med 2001, 20:2865-2884.
Reitsma JB, Glas AS, Rutjes AWS, Scholten RJPM, Bossuyt PM, Zwinderman AH: Bivariate analysis of sensitivity and specificity produces informative summary measures in diagnostic reviews.
Journal of Clinical Epidemiology 2005, 58:982-990.
Walter SD: Properties of the summary receiver operating
characteristic (SROC) curve for diagnostic test data. Stat
Med 2002, 21:1237-1256.
DerSimonian R, Laird N: Meta-analysis in clinical trials. Control
Clin Trials 1986, 7:177-188.
Gupta JK, Chien PF, Voit D, Clark TJ, Khan KS: Ultrasonographic
endometrial thickness for diagnosing endometrial pathology
in women with postmenopausal bleeding: a meta-analysis.
Acta Obstet Gynecol Scand 2002, 81:799-816.
Khan KS, Kunz R, Kleijnen J, Antes G: Case study 4: Reviewing
evidence on test accuracy. In Systematic Review to Support Evidence-based Medicine 2003 edition. London, The Royal Society of
Medicine; 2003:109-119.
Morgan M, Kalantri S, Flores L, Pai M: A commercial line probe
assay for the rapid detection of rifampicin resistance in
Mycobacterium tuberculosis: a systematic review and metaanalysis. BMC Infectious Diseases 2005, 5:62.
Flores L, Pai M, Colford JM, Riley LW: In-house nucleic acid
amplification tests for the detection of Mycobacterium
tuberculosis in sputum specimens: meta-analysis and metaregression. BMC Microbiol 2005, 5:55.
Gisbert J, Abraira V: Accuracy of Helicobacter pylori Diagnostic Tests in Patients with Bleeding Peptic Ulcer: A Systematic Review and Meta-analysis.
The American Journal of
Gastroenterology 2006, 101:848-863.
Shiga T, Wajima Z, Inoue T, Sakamoto A: Predicting difficult intubation in apparently normal patients: a meta-analysis of bedside screening test performance.
Anesthesiology 2005,
103:429-437.
Zijlstra JM, van der Werf G, Hoekstra OS, Hooft L, Huijgens PC: Ffluoro-deoxyglucose positron emission tomography for posttreatment evaluation of malignant lymphoma: a systematic
review. Haematologica 2006, 91:522-9.
Goodacre S, Sutton AJ, Sampson FC: Meta-Analysis: The Value of
Clinical Assessment in the Diagnosis of Deep Venous
Thrombosis. Annals of Internal Medicine 2005, 143:129-139.

Pre-publication history
The pre-publication history for this paper can be accessed
here:
http://www.biomedcentral.com/1471-2288/6/31/prepub
Page 12 of 12
(page number not for citation purposes)