ANZ J. Surg.

2003; 73: 712716

ORIGINAL ARTICLE
ORIGINAL ARTICLE

METHODOLOGICAL INDEX FOR NON-RANDOMIZED STUDIES (MINORS ):

DEVELOPMENT AND VALIDATION OF A NEW INSTRUMENT

KAREM SLIM,* EMILE NINI,* DAMIEN FORESTIER,* FABRICE KWIATKOWSKI, YVES PANIS
AND JACQUES CHIPPONI*
*Department of General and Digestive Surgery, Htel-Dieu, Clermont-Ferrand, Department of Statistics, Centre Jean-Perrin
Clermont-Ferrand and Department of Digestive Surgery, Hpital Lariboisire, Paris, France

Background: Because of specific methodological difficulties in conducting randomized trials, surgical research remains depen-
dent predominantly on observational or non-randomized studies. Few validated instruments are available to determine the methodol-
ogical quality of such studies either from the readers perspective or for the purpose of meta-analysis. The aim of the present study
was to develop and validate such an instrument.
Methods: After an initial conceptualization phase of a methodological index for non-randomized studies (MINORS), a list of
12 potential items was sent to 100 experts from different surgical specialities for evaluation and was also assessed by 10 clinical
methodologists. Subsequent testing involved the assessment of inter-reviewer agreement, test-retest reliability at 2 months, internal
consistency reliability and external validity.
Results: The final version of MINORS contained 12 items, the first eight being specifically for non-comparative studies. Reliabil-
ity was established on the basis of good inter-reviewer agreement, high test-retest reliability by the -coefficient and good internal
consistency by a high Cronbachs -coefficient. External validity was established in terms of the ability of MINORS to identify
excellent trials.
Conclusions: MINORS is a valid instrument designed to assess the methodological quality of non-randomized surgical studies,
whether comparative or non-comparative. The next step will be to determine its external validity when used in a large number of
studies and to compare it with other existing instruments.

Key words: comparative study, methodology index, non-randomized study.

Abbreviation: MINORS, methodological index for non-randomized studies.

INTRODUCTION both randomized and non-randomized studies without distinc-

tion.6 The aim of the present study was to develop and validate a
Although surgeons are now conducting an increasing number of
methodological index for non-randomized studies (MINORS)
randomized trials,1 most of the available evidence in surgery
which could be used by readers, manuscript reviewers or journal
comes from non-randomized studies, both comparative and non-
editors to assess the quality of such studies.
comparative. Indeed surgical research remains an example of a
situation where randomization is not always possible or feasible.2
Beyond large randomized trials, systematic reviews are an impor- METHODS
tant way to answer questions in surgery. However, the systematic
Conceptualization phase
review or meta-analysis of studies other than randomized trials
may be difficult because combining the results of observational After reviewing the literature on quality assessment of randomized
studies of heterogeneous quality could be highly biased. trials and discussing the particular features of non-randomized
Observational studies include comparative studies such as studies, a panel of eight practising surgeons selected 12 items to
case-control and cohort designs, and patient series which may or be considered for inclusion in MINORS. These items were chosen
may not involve comparisons between two or more groups. because of their ability to characterize the methodological and sci-
Several papers have discussed the methodology of meta- entific value of published articles. Seven items were selected for
analyses of observational studies3,4 and checklists have been pro- assessment of non-comparative studies and five for use with com-
posed but not formally validated.5 Downs and Black used clini- parative studies. The list of 12 items was then sent to 100 surgeons
metric criteria to develop a checklist which was applicable to throughout France who had clinical research expertise in different
specialities, including digestive, cardiovascular and thoracic sur-
gery, gynaecology, otorhinolaryngology, orthopaedics, urology,
neurosurgery, and ophthalmology. They were asked to score the
K. Slim MD, FACS; E. Nini MD; D. Forestier MD; F. Kwiatkowski PhD; ability of each item to assess the quality of a given study using a
Y. Panis MD, PhD; J. Chipponi MD, PhD. 7-point-scale, according to the method proposed by Oxman and
Guyatt.7 The mean score for each item was then compared with
Correspondence: K. Slim, Department of General and Digestive Surgery, that of every other item to see whether there were any significant
Htel-Dieu Boulevard Leon Malfreyt, F-63058 Clermont-Ferrand, France.
differences. Subsequently each item was scored from 0 to 2;
Email: [email protected]
0 indicating that it was not reported in the article evaluated, 1 indi-
Accepted for publication 13 May 2003. cating that it was reported but inadequately, and 2 indicating that
METHODOLOGICAL INDEX FOR NON-RANDOMIZED STUDIES 713

it was reported adequately. The form also included a section RESULTS

allowing the surgeons to suggest additional items.
Content and face validity

Assessment of face validity and content validity Expert phase

Ninety of the 100 experts returned a completed form. Table 1
To determine whether MINORS items appeared appropriate and summarizes the scores of the 12 items included in the first version
whether they covered all important considerations relevant to the of MINORS. No item was scored less than five. Furthermore
methodology of non-randomized studies, a revised list of items there was no difference between the different specialities. The
was sent to 10 French clinical methodologists for assessment on experts suggested no additional methodological items apart from
13 credibility criteria according to the method proposed by a modification of item 11. As a result a supplementary sentence
Feinstein.8 was incorporated in that item in the revised version of MINORS
relating to the size of non-comparative studies. Item 11 thus
Clinimetric testing of MINORS became relevant to both comparative and non-comparative
studies. The revised version of MINORS included 12 items: the
Inter-reviewer agreement first subscale of eight items related to non-comparative studies
To test the consistency of MINORS between reviewers, a random whereas all 12 items were relevant to comparative studies
sample of published non-randomized studies, both comparative (Table 2).
and non-comparative, was selected from among several special-
ties. For this purpose a Medline search was undertaken using the Revision phase
MeSH surgery and limits by publication type (clinical trial not Because there was no statistical difference between the mean
randomized controlled trial) for the year 2001. A numerical list of item scores as evaluated by the experts, the items were not
original articles was then established and 80 articles were weighted and the scoring was simplified to a 3-point scale from
selected randomly. The title, authors names, institutional affilia- 0 to 2. If one considers that MINORS involves eight items for
tion and journal identity were removed. These articles were then non-comparative studies and 12 items for comparative studies
assessed by two independent reviewers with different methodo- and that the maximum item score is 2, the ideal global score
logical expertise (one junior and one senior surgeon) using the would be 16 for the non-comparative studies and 24 for the
revised version of MINORS. comparative studies.

Test-retest reliability Methodologist phase

Two months after the first assessment, a randomly selected All 10 methodologists completed their assessment and scored the
sample of 30 articles was scored again by the junior surgeon final version favourably, all item mean scores being above 4.5 on
without reference to his first assessment. a 7-point scale (Table 3).

Internal consistency
This evaluation indicated whether the items were related to one
another and worked together in a similar manner in assessing the
quality of articles.
Table 1. Assessment of items in the rst version of MINORS by
90 experts in several surgical specialities using a scale from 0 to 7
Validity
The power of MINORS to differentiate between excellent, fair Item Median Mean (SD)
or poor studies was examined by selecting a random sample
of 15 excellent randomized controlled trials. These articles were 1. A stated aim of the study 7 6.6 (0.7)
chosen as the gold standard against which to assess the external 2. Inclusion of consecutive 6 5.8 (1.1)
patients
validity of MINORS on the basis that they had all been
3. Prospective collection of data 6 5.5 (1.2)
published in three major journals which had adopted the 4. Endpoint appropriate to the 6 6.3 (0.8)
CONSORT Statement,9 (namely British Medical Journal, study aim
British Journal of Surgery and The Lancet). These articles were 5. Unbiased evaluation of 5 5.4 (1.2)
then scored according to MINORS and the results compared endpoints
with a selected group of the 15 best-scored comparative studies 6. Follow-up period appropriate 6 6.2 (0.8)
from the sample of 80 described previously. The reviewer was to the major endpoint
blinded as to the source of the 15 randomized trials. 7. Loss to follow up not 6 5.5 (1.2)
exceeding 5%

Statistical analyses And in the case of comparative studies

8. A control group having the 6 6.0 (1.1)
Agreement between reviewers was measured by the -coefficient gold standard intervention
(unweighted model) with a value greater than 0.4 being accepted 9. Contemporary groups 6 5.5 (1.4)
as satisfactory.10 Global scores were obtained by summing all the 10. Baseline equivalence of groups 6 6.1 (0.9)
item scores. Results were expressed as means (standard devia- 11. Prospective calculation of the 6 5.5 (1.3)
tions). The matched pairs t-test was used to compare mean global sample size
12. Statistical analyses adapted to 6 6.3 (0.8)
scores between reviewers. Internal consistency was assessed by
the study design
the calculation of Cronbachs -coefficient.11 A value of P < 0.05
was considered statistically significant. SD, standard deviation.
714 SLIM ET AL.

Inter-reviewer agreement, test-retest reliability and internal DISCUSSION

consistency reliability
This index for the assessment of non-randomized studies was
There were 26 comparative and 54 non-comparative studies in developed by a group of surgeons because of the problems faced
the random sample assessed by the two reviewers. Table 4 sum- by clinicians as to the lack of randomized surgical trials and
marizes the correlation between the scores of the reviewers. the large number of observational studies in surgery. To apply the
Agreement between the reviewers was considered satisfactory for principles of evidence-based medicine to clinical practice
all items. The mean global scores on a scale from 0 to 24 were, requires a method for assessing the quality of published data.
respectively, 13.93 (0.35) for the junior surgeon and 12.98 (0.54)
for the senior surgeon. This difference was statistically signifi-
cant (P < 107) but corresponds to only 0.95 of a global score
point. The mean global scores, which ranged between 0 and 20 in Table 3. Credibility criteria assessed by 10 clinical methodologists
this agreement assessment, did not differ significantly between on a 7-point scale
the comparative and non-comparative studies (P = 0.11).
The assessment of test-retest reliability showed a satisfactory Criterion Mean SD (range)
correlation between the original and repeated scoring after a 2-month
interval. The mean global score decreased significantly from 13.91 1. Wide applicability 5.5 0.5 (56)
2. Use by various groups 4.8 0.8 (46)
(3.3) at the first test to 12.28 (3.6) at the second (P < 0.0001). 3. Clarity and simplicity 5.1 0.8 (46)
The internal consistency reliability of MINORS was high with a 4. Adequate instructions 5.0 1.0 (47)
global -value of 0.73. This demonstrated that all items worked 5. Information available 4.9 1.2 (37)
in a complementary and coherent manner. 6. Need for subjective decision 4.7 0.9 (47)
7. Likelihood of bias 4.8 1.2 (37)
8. Single domain 5.1 0.9 (47)
Validity 9. Redundant items 5.6 0.8 (47)
The 15 gold-standard randomized trials had a mean global score 10. Comprehensiveness 5.1 0.7 (46)
of 23.1. The comparison between the score of these randomized 11. Item weights 5.4 1.0 (37)
trials and that of the 15 best comparative non-randomized studies 12. Number of response options 5.4 1.1 (47)
13. Discrimination power 5.1 0.5 (46)
(19.8) showed a significant difference (P = 0.00001) in favour of
the randomized trials. SD, standard deviation.

Table 2. The revised and validated version of MINORS

Methodological items for non-randomized studies Score

1. A clearly stated aim: the question addressed should be precise and relevant in the light of available literature
2. Inclusion of consecutive patients: all patients potentially t for inclusion (satisfying the criteria for inclusion) have been
included in the study during the study period (no exclusion or details about the reasons for exclusion)
3. Prospective collection of data: data were collected according to a protocol established before the beginning of the study
4. Endpoints appropriate to the aim of the study: unambiguous explanation of the criteria used to evaluate the main outcome
which should be in accordance with the question addressed by the study. Also, the endpoints should be assessed on an
intention-to-treat basis.
5. Unbiased assessment of the study endpoint: blind evaluation of objective endpoints and double-blind evaluation of subjective
endpoints. Otherwise the reasons for not blinding should be stated
6. Follow-up period appropriate to the aim of the study: the follow-up should be sufciently long to allow the assessment of
the main endpoint and possible adverse events
7. Loss to follow up less than 5%: all patients should be included in the follow up. Otherwise, the proportion lost to follow up
should not exceed the proportion experiencing the major endpoint
8. Prospective calculation of the study size: information of the size of detectable difference of interest with a calculation of
95% condence interval, according to the expected incidence of the outcome event, and information about the level for
statistical signicance and estimates of power when comparing the outcomes
Additional criteria in the case of comparative study
9. An adequate control group: having a gold standard diagnostic test or therapeutic intervention recognized as the optimal
intervention according to the available published data
10. Contemporary groups: control and studied group should be managed during the same time period (no historical comparison)
11. Baseline equivalence of groups: the groups should be similar regarding the criteria other than the studied endpoints. Absence
of confounding factors that could bias the interpretation of the results
12. Adequate statistical analyses: whether the statistics were in accordance with the type of study with calculation of condence
intervals or relative risk
The items are scored 0 (not reported), 1 (reported but inadequate) or 2 (reported and adequate). The global ideal score being 16 for non-comparative studies
and 24 for comparative studies.
METHODOLOGICAL INDEX FOR NON-RANDOMIZED STUDIES 715

Table 4. List of 12 items of the denitive MINORS. Inter-reviewer correlation on a random sample of 80 articles and test-retest reliability on
a random sample of 30 articles.
Methodological item for non-randomized studies -coefcient for inter-reviewer -coefcient for test-re-test
agreement (SD) reliability (SD)

1. A clearly stated aim 0.87 (0.07) 0.89 (0.11

2. Inclusion of consecutive patients 0.78 (0.06) 0.83 (0.09)
3. Prospective collection of data 0.79 (0.06) 0.82 (0.09)
4. Endpoints appropriate to the aim of the study 0.56 (0.09) 0.76 (0.12)
5. Unbiased assessment of the study endpoint 0.61 (0.08) 0.61 (0.13)
6. Follow-up period appropriate to the aim of the study 0.61 (0.08) 0.59 (0.11)
7. Loss to follow up less than 5% 0.69 (0.08) 0.74 (0.12)
8. Prospective calculation of the study size 1.00 1.00
Additional criteria in the case of comparative studies
9. An adequate control group 0.86 (0.09) 1.00
10. Contemporary groups 0.79 (0.14) 0.61 (0.31)
11. Baseline equivalence of groups 0.87 (0.09) 1.00
12. Adequate statistical analyses 0.66 (0.14) 0.75 (0.22)
A -coefcient of >0.4 was considered satisfactory.

This is an important consideration for the consumers of clinical surgery. Currently, however, there is no sound evidence for the
research. Our initial aim was to develop and validate an index differential weighting of items in methodological indices or
which would be simple to use both by readers of published checklists for non-randomized studies.
articles and reviewers of manuscripts submitted for publication, Downs and Black6 reported a checklist applicable to both rand-
and be of sufficient sensitivity for use in meta-analysis of non- omized and non-randomized trials. It involved 27 items concerning
randomized studies. To achieve this we followed the recognized external validity, bias, confounding factors, statistical power and
principles of scale construction12 using a rigorous methodology. reporting; however, the number of items and differences in scoring
The results of the present study show clearly that the instrument systems between items increased complexity and user burden.
we have developed has good reliability, internal consistency and Several items were related to reporting and thus were not directly
validity. The high response rate from experts and the limited concerned with the methodological quality of a study. Also in their
number of items used, suggest that MINORS is easy to apply. Its study, the period between the test and re-test was only 2 weeks and
simplicity and objectivity is also demonstrated by its acceptabil- the reviewers were similar to one-another in their level of method-
ity to surgeons having sound methodological expertise. Although ological skill. Furthermore their instrument was a checklist and
the difference between the scores of the senior and the junior was not developed as an index for scoring studies.
reviewers was statistically significant, its actual relevance was An important aspect of MINORS is its external validity;
low as the difference did not exceed 1 point. that is, its ability to identify high quality studies, which was
Similarly, the assessment of test-retest reliability showed a established by comparison with the current standard for rand-
good correlation over an interval of 2 months. The reviewer omized trials, namely the CONSORT Statement. Since MINORS
scored perhaps more severely on the second occasion, which sug- does not differentiate between randomized and non-randomized
gests greater expertise with further experience, but the difference studies and includes several items derived from indices focusing
was too small (1.6) to be important. Nevertheless this feature may on the quality of randomized trials, the fact that a given study has
need to be investigated further. a randomized design is not sufficient to achieve a high score.
Instead of weighting, we chose to score the items from 0 to 2 MINORS was not developed specifically to assess the quality of
according to whether they were reported or not and adequate or randomized trials; however, we considered the randomized trial
not. Weighting of items requires further investigation as we have to be the best example of comparative studies and assumed that
no gold standard method to evaluate the relative importance of a MINORS should be able to distinguish between different com-
given methodological item. In the light of the available literature, parative studies. MINORS satisfied that expectation and clearly
the most appropriate method of weighting would be based on confirmed that a good randomized trial scores higher than a good
consensus development among experienced epidemiologists non-randomized comparative study. The ability of MINORS to
before designing a large study to validate their conclusions. Few recognize the poor or fair quality of non-comparative studies is
attempts have been made to estimate the respective values of suggested in our study, but this needs to be further evaluated by
some methodological items.13 Furthermore the most significant comparison with the Downs and Black checklist.6 This com-
findings regarding the weighting of items have been specifically parison will be the subject of a future study to develop a reliable
related to randomized double blind studies. One could assume standardized instrument for assessing the quality of non-
that the rationale for weighting in randomized trials can be randomized studies, especially for the purposes of meta-analysis.
extrapolated to non-randomized studies. However this needs to Nevertheless, as with randomized trials14 for which there is no
be confirmed by further investigations, especially in the field of gold standard, it is possible that any newly proposed instrument
surgery. Furthermore, the item weights could differ according to might have internal flaws. An ideal index should be highly sensi-
the type of study. For example unbiased evaluation of endpoints tive (by increasing the number of items) and applicable in daily
is important for functional disorders whereas the length of follow practice (by minimizing user burden). This remains the challenge
up and loss to follow up are important for hernia or cancer for epidemiologists and research in this field is in its infancy.
716 SLIM ET AL.

MINORS in our opinion has two important attributes. First, 4. Bhandari M, Morrow F, Kulkarni AV, Tornetta P. Meta-analyses
its simplicity in comprising only 12 items that are readily usable in orthopaedic surgery. A systematic review of their methodolo-
by both readers and researchers and second, its reliability, as gies. J. Bone Joint Surg. Am. 2001; 83: 1524.
demonstrated by clinimetric testing. Our aim now is to use 5. Khan KS, ter Riet G, Popay J, Nixon J, Kleijnen J. Study quality
MINORS in several more studies designed to evaluate the meth- assessment. In: Undertaking systematic reviews of research on
effectiveness. CRDs guidance for those carrying out or commis-
odology of non-randomized studies. Only the repeated use of sioning reviews CRD Report 4, 2nd edn. York: York Publishing
such an instrument can confirm the present preliminary clini- Services; March 2001.
metric validation. 6. Downs SH, Black N. The feasibility of creating a checklist for
the assessment of the methodological quality both of randomised
ACKNOWLEDGEMENTS and non-randomised studies of health care interventions. J. Epi-
demiol. Community Health 1998; 52: 37784.
We thank the following participants for their help in the phases
7. Oxman AD, Guyatt GH. Validation of an index of the quality of
of conceptualization and validation: L. Audig (from the Non- review articles. J. Clin. Epidemiol. 1991; 44: 12718.
randomized Study Method Group of the Cochrane Collabora- 8. Feinstein AR. Clinimetrics. New Haven: Yale University Press,
tion), E. Albuisson, J. E. Bazin, F. Blanchard, G. Borges Da 1987.
Silva, S. Bouee, D. Chopin, B. Dousset, C. Dziri, F. Fagnani, 9. Moher D, Schulz KF, Altman DG for the CONSORT Group.
P-L Fagniez, L. Gerbaud, A. Kramar, F. Lacaine, B. Millat, CONSORT statement: revised recommendations for improving
E. Monnet, P. Perez, A. Perillat, T. Perniceni, L. R. Salmi and the the quality of reports of parallel-group randomised trials. Lancet
surgeons who returned the completed form during the conceptu- 2001; 357: 11914.
alization phase of MINORS. 10. King TS, Chinchilli VM. A generalized concordance correlation
coefficient for continuous and categorical data. Stat. Med. 2001;
20: 213147.
REFERENCES
11. Bland JM, Altman DG. Cronbachs alpha. BMJ 1997; 314: 572.
1. Slim K, Haugh M, Fagniez P-L, Pezet D, Chipponi J. Ten-year 12. Bland JM, Altman DG. Validating scales and indexes. BMJ
audit of randomised trials in digestive surgery from Europe. 2002; 324: 6067.
Br. J. Surg. 2000; 87: 15856. 13. Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evi-
2. McCulloch PM, Taylor I, Sasako M, Lovett B, Griffin D. Ran- dence of bias. Dimensions of methodological quality associated
domised trials in surgery: problems and possible solutions. BMJ with estimates of treatment effects in controlled trials. JAMA
2002; 324: 144851. 1995; 273: 40812.
3. Stroup DF, Berlin JA, Morton SC et al. Meta-analysis of obser- 14. Jni P, Witschi A, Bloch R, Egger M. The hazards of scoring
vational studies in epidemiology. A proposal for reporting. the quality of clinical trials for meta-analysis. JAMA 1999; 282:
JAMA 2000; 283: 200812. 105460.