Guidelines for performing

Systematic Literature Reviews

in Software Engineering
Barbara Kitchenham, und Stuart Charters. EBSE 2007-001.
Keele University and Durham University Joint Report, (2007)

Source: „Guidelines for performing Systematic Literature Reviews in SE“, Kitchenham et al., 2007
0.4
Executive
Executive Summary
summary
The objective of this report is to propose comprehensive guidelines for systematic
literature reviews appropriate for software engineering researchers, including PhD
students. A systematic literature review is a means of evaluating and interpreting all
available research relevant to a particular research question, topic area, or
phenomenon of interest. Systematic reviews aim to present a fair evaluation of a
research topic by using a trustworthy, rigorous, and auditable methodology.

The guidelines presented in this report were derived from three existing guidelines
used by medical researchers, two books produced by researchers with social science
backgrounds and discussions with researchers from other disciplines who are involved
in evidence-based practice. The guidelines have been adapted to reflect the specific
problems of software engineering research.

The guidelines cover three phases of a systematic literature review: planning the
review, conducting the review and reporting the review. They provide a relatively
high level description. They do not consider the impact of the research questions on
the review procedures, nor do they specify in detail the mechanisms needed to
perform meta-analysis.

0.5 Glossary
Source: „Guidelines for performing Systematic Literature Reviews in SE“, Kitchenham et al., 2007
 Examples of SLRs
R.F. Barcelos, G.H. Travassos, Evaluation approaches for software architectural documents: a
systematic review, in: Ibero-American Workshop on Requirements Engineering and
Software Environments (IDEAS), La Plata, Argentina, 2006.

T. Dyba, V.B. Kampenes, D.I.K. Sjøberg, A systematic review of statistical power in software
engineering experiments, Information and Software Technology 48 (8) (2006) 745–755.

D. Galin, M. Avrahami, Do SQA programs work – CMM works. A meta analysis, IEEE
International Conference on Software – Science, Technology and Engineering (2005).

D. Galin, M. Avrahami, Are CMM program investments beneficial? Analyzing past studies,
IEEE Software 23 (6) (2006) 81–87.

J.E. Hannay, D.I.K. Sjøberg, T. Dybå, A systematic review of theory use in software
engineering experiments, IEEE Transactions on SE 33 (2) (2007) 87– 107.

M. Jørgensen, Estimation of software development work effort: evidence on expert judgement
and formal models, International Journal of Forecasting 3 (3) (2007) 449–462.

N. Juristo, A.M. Moreno, S. Vegas, Reviewing 25 years of testing technique experiments,
Empirical Software Engineering Journal (1–2) (2004) 7–44.

Source: „Guidelines for performing Systematic Literature Reviews in SE“, Kitchenham et al., 2007
the review procedures, nor do they specify in detail the mechanisms needed to
perform meta-analysis.

0.5 Glossary Glossary 1/2

Meta-analysis. A form of secondary study where research synthesis is based on
quantitative statistical methods.

Primary study. (In the context of evidence) An empirical study investigating a

specific research question.

Secondary study. A study that reviews all the primary studies relating to a specific
research question with the aim of integrating/synthesising evidence related to a
specific research question.

Sensitivity analysis. An analysis procedure aimed at assessing whether the results of a

systematic literature review or a meta-analysis are unduly influenced by a small
number of studies. Sensitivity analysis methods involve assessing the impact of high
leverage studies (e.g. large studies or studies with atypical results), and ensuring that
overall results of a systematic literature remain the same if low quality studies (or
high quality) studies are omitted from the analysis, or analysed separately.

Systematic literature review (also referred to as a systematic review). A form of

secondary study that uses a well-defined methodology to identify, analyse and
interpret all available evidence related to a specific research question in a way that is
unbiased and (to a degree) repeatable.
Source: „Guidelines for performing Systematic Literature Reviews in SE“, Kitchenham et al., 2007
high quality) studies are omitted from the analysis, or analysed separately.

Glossary 2/2
Systematic literature review (also referred to as a systematic review). A form of
secondary study that uses a well-defined methodology to identify, analyse and
interpret all available evidence related to a specific research question in a way that is
unbiased and (to a degree) repeatable.

Systematic review protocol. A plan that describes the conduct of a proposed

systematic literature review.

Systematic mapping study (also referred to as a scoping study). A broad review of

primary studies in a specific topic area that aims to identify what evidence is available
on the topic.
vi
Tertiary study (also called a tertiary review). A review of secondary studies related to
the same research question.

Source: „Guidelines for performing Systematic Literature Reviews in SE“, Kitchenham et al., 2007
A systematic literature review (often referred to as a systematic review) is a means of
identifying, evaluating and interpreting all available research relevant to a particular
Reasons for performing a SLR
research question, or topic area, or phenomenon of interest. Individual studies
contributing to a systematic review are called primary studies; a systematic review is
a form of secondary study.

2.1 Reasons for Performing Systematic Literature Reviews

There are many reasons for undertaking a systematic literature review. The most
common reasons are:
To summarise the existing evidence concerning a treatment or technology e.g. to
summarise the empirical evidence of the benefits and limitations of a specific
agile method.
To identify any gaps in current research in order to suggest areas for further
investigation.
To provide a framework/background in order to appropriately position new
research activities.

However, systematic literature reviews can also be undertaken to examine the extent
to which empirical evidence supports/contradicts theoretical hypotheses, or even to
assist the generation of new hypotheses (see for example [14]).

2.2 The Importance of Systematic Literature Reviews

Source: „Guidelines for performing Systematic Literature Reviews in SE“, Kitchenham et al., 2007
 To provide a framework/background in order to appropriately position new
research activities.
The importance of SLRs
However, systematic literature reviews can also be undertaken to examine the extent
to which empirical evidence supports/contradicts theoretical hypotheses, or even to
assist the generation of new hypotheses (see for example [14]).

2.2 The Importance of Systematic Literature Reviews

Most research starts with a literature review of some sort. However, unless a literature
review is thorough and fair, it is of little scientific value. This is the main rationale for
undertaking systematic reviews. A systematic review synthesises existing work in a
manner that is fair and seen to be fair. For example, systematic reviews must be
undertaken in accordance with a predefined search strategy. The search strategy must
allow the completeness of the search to be assessed. In particular, researchers
performing a systematic review must make every effort to identify and report research
that does not support their preferred research hypothesis as well as identifying and
reporting research that supports it.

"Indeed, one of my major complaints about the computer field is that

whereas Newton could say, "If I have seen a little farther than
others, it is because I have stood on the shoulders of giants," I am
forced to say, "Today we stand on each other's feet." Perhaps the
central problem we face in all of computer science is how we are to
get to the situation where we build on top of the work of others rather
than redoing so much
Source: „Guidelines of it inSystematic
for performing a trivially different
Literature Reviews inway. Scienceet is
SE“, Kitchenham al., 2007
Systematic literature reviews in all disciplines allow us to stand on the shoulders of
giants and in computing, allow us to get off each others’ feet.
Advantage and disadvantages of SLRs
2.3 Advantages and disadvantages

The advantages of systematic literature reviews are that:

The well-defined methodology makes it less likely that the results of the
literature are biased, although it does not protect against publication bias in the
primary studies.
They can provide information about the effects of some phenomenon across a
wide range of settings and empirical methods. If studies give consistent results,
systematic reviews provide evidence that the phenomenon is robust and
transferable. If the studies give inconsistent results, sources of variation can be
studied.
In the case of quantitative studies, it is possible to combine data using meta-
analytic techniques. This increases the likelihood of detecting real effects that
individual smaller studies are unable to detect.

The major disadvantage of systematic literature reviews is that they require

considerably more effort than traditional literature reviews. In addition, increased
power for meta-analysis can also be a disadvantage, since it is possible to detect small
biases as well as true effects.

2.4 Features of Systematic Literature Reviews

Source: „Guidelines for performing Systematic Literature Reviews in SE“, Kitchenham et al., 2007
The major disadvantage of systematic literature reviews is that they require
considerably more effort than traditional literature reviews. In addition, increased
Features of SLRs
power for meta-analysis can also be a disadvantage, since it is possible to detect small
biases as well as true effects.

2.4 Features of Systematic Literature Reviews

Some of the features that differentiate a systematic review from a conventional expert
literature review are:
Systematic reviews start by defining a review protocol that specifies the research
question being addressed and the methods that will be used to perform the review.
Systematic reviews are based on a defined search strategy that aims to detect as
much of the relevant literature as possible.
Systematic reviews document their search strategy so that readers can assess their
rigour and the completeness and repeatability of the process (bearing in mind that
searches of digital libraries are almost impossible to replicate).
Systematic reviews require explicit inclusion and exclusion criteria to assess each
potential primary study.
Systematic reviews specify the information to be obtained from each primary
study including quality criteria by which to evaluate each primary study.
A systematic review is a prerequisite for quantitative meta-analysis.

2.5 Other Types of Review

There are two other types of review that complement systematic literature reviews:
Source: „Guidelines for performing Systematic Literature Reviews in SE“, Kitchenham et al., 2007
 medicine. This similarity is due to experimental practices, subject types and blinding
procedures. Within Software Engineering it is difficult to conduct randomised
controlled trials or to undertake
Comparing Softwaredouble Engineering
blinding. In addition, experimental
human expertise and the
human subject all affect the outcome of experiments.
methodology with that of other disciplines
Table 1 Comparing Software Engineering experimental methodology with that
of other disciplines
Comparison with SE (1 is perfect agreement, 0 is
Discipline
complete disagreement)
Nursing & Midwifery 0.83
Primary Care 0.33
Organic Chemistry 0.83
Empirical Psychology 0.66
Clinical Medicine 0.17
Education 0.83

These factors mean that software engineering is significantly different from the
traditional
These factorsmedical
mean arena in which systematic
that software engineeringreviews were first developed.
is significantly For this
different from the
traditional medical arena in which systematic reviews were first developed.
5

Source: „Guidelines for performing Systematic Literature Reviews in SE“, Kitchenham et al., 2007
 The review process

• Planning the review

• Conducting the review

• Reporting the review

Source: „Guidelines for performing Systematic Literature Reviews in SE“, Kitchenham et al., 2007
 The review process
• Planning the review

• Identification of the need for a review

• (Commissioning a review)

• Specifying the research questions

• Developing a review protocol

• (Evaluating the review protocol)

• Conducting the review

• Reporting the review

Source: „Guidelines for performing Systematic Literature Reviews in SE“, Kitchenham et al., 2007
 The review process
• Planning the review

• Conducting the review

• Identification of research

• Selection of primary research

• Study quality assessment

• Data extraction and monitoring

• Data synthesis

• Reporting the review

Source: „Guidelines for performing Systematic Literature Reviews in SE“, Kitchenham et al., 2007
 The review process
• Planning the review

• Conducting the review

• Reporting the review

• Specifying dissemination mechanisms

• Formatting the main report

• Evaluating the report

Source: „Guidelines for performing Systematic Literature Reviews in SE“, Kitchenham et al., 2007
The need for a systematic review arises from the requirement of researchers to

The need for a review

summarise all existing information about some phenomenon in a thorough and
unbiased manner. This may be in order to draw more general conclusions about some
phenomenon than is possible from individual studies, or may be undertaken as a
prelude to further research activities.

Examples

Kitchenham et al. [21] argued that accurate cost estimation is important for the software
industry; that accurate cost estimation models rely on past project data; that many companies
cannot collect enough data to construct their own models. Thus, it is important to know
whether models developed from data repositories can be used to predict costs in a specific
company. They noted that a number of studies have addressed that issue but have come to
different conclusions. They concluded that it is necessary to determine whether, or under
what conditions, models derived from data repositories can support estimation in a specific
company.

Jørgensen [17] pointed out in spite of the fact that most software cost estimation research
concentrates on formal cost estimation models and that a large number of IT managers know
about tools that implement formal models, most industrial cost estimation is based on expert
judgement. He argued that researchers need to know whether software professionals are
simply irrational, or whether expert judgement is just as accurate as formal models or has
other advantages that make it more acceptable than formal models.

In both cases the authors had undertaken research in the topic area and had first hand
knowledge of the research issues.
Source: „Guidelines for performing Systematic Literature Reviews in SE“, Kitchenham et al., 2007
advisory group to ensure the review remains focused and relevant in the context.

The commissioning phase of a systematic review is not required for a research team

The research questions

undertaking a review for their own needs or for one being undertaken by a PhD
student. If the commissioning stage is not undertaken then the dissemination strategy
should be incorporated into the review protocol. As yet, there are no examples of
commissioned SLRs in the software engineering domain.

5.3 The Research Question(s)

Specifying the research questions is the most important part of any systematic review.
The review questions drive the entire systematic review methodology:
The search process must identify primary studies that address the research
questions.
The data extraction process must extract the data items needed to answer the
questions.
The data analysis process must synthesise the data in such a way that the
questions can be answered.

5.3.1 Question Types

The most important activity during planning is to formulate the research question(s).
The Australian NHMR Guidelines [1] identify six types of health care questions that
can be addressed by systematic reviews:
1. Assessing the effect of intervention.
2. Assessing the frequency or rate of a condition or disease.
3. Determining the performance
Source: „Guidelines for performing of a diagnostic
Systematic test. in SE“, Kitchenham et al., 2007
Literature Reviews
 The search process must identify primary studies that address the research
questions.
The data extraction process must extract the data items needed to answer the
Question types in health care
questions.
The data analysis process must synthesise the data in such a way that the
questions can be answered.

5.3.1 Question Types

The most important activity during planning is to formulate the research question(s).
The Australian NHMR Guidelines [1] identify six types of health care questions that
can be addressed by systematic reviews:
1. Assessing the effect of intervention.
2. Assessing the frequency or rate of a condition or disease.
3. Determining the performance of a diagnostic test.
4. Identifying aetiology and risk factors.
5. Identifying whether a condition can be predicted.
6. Assessing the economic value of an intervention or procedure.

In software engineering, it is not clear what the equivalent of a diagnostic test would
be, but the other questions can be adapted to software engineering issues as follows:
Assessing the effect of a software engineering technology.
Assessing the frequency or rate of a project development factor such as the
adoption of a technology, or the frequency or rate of project success or failure.
Identifying cost and risk factors associated with a technology.
Identifying the impact of technologies on reliability, performance and cost
Source: „Guidelines for performing Systematic Literature Reviews in SE“, Kitchenham et al., 2007
The Australian NHMR Guidelines [1] identify six types of health care questions that
can be addressed by systematic reviews:
1. Assessing the effect of intervention.
Question types in software engineering
2. Assessing the frequency or rate of a condition or disease.
3. Determining the performance of a diagnostic test.
4. Identifying aetiology and risk factors.
5. Identifying whether a condition can be predicted.
6. Assessing the economic value of an intervention or procedure.

In software engineering, it is not clear what the equivalent of a diagnostic test would
be, but the other questions can be adapted to software engineering issues as follows:
Assessing the effect of a software engineering technology.
Assessing the frequency or rate of a project development factor such as the
adoption of a technology, or the frequency or rate of project success or failure.
Identifying cost and risk factors associated with a technology.
Identifying the impact of technologies on reliability, performance and cost
models.
Cost benefit analysis of employing specific software development technologies or
software applications.

Medical guidelines often provide different guidelines and procedures for different 9
types of question. This document does not go to this level of detail.

The critical issue in any systematic review is to ask the right question. In this context,
the right question is usually one that:
Source: „Guidelines for performing Systematic Literature Reviews in SE“, Kitchenham et al., 2007
 Cost benefit analysis of employing specific software development technologies or
Asking the right question
software applications.

Medical guidelines often provide different guidelines and procedures for different
types of question. This document does not go to this level of detail.

The critical issue in any systematic review is to ask the right question. In this context,
the right question is usually one that:
Is meaningful and important to practitioners as well as researchers. For example,
researchers might be interested in whether a specific analysis technique leads to a
significantly more accurate estimate of remaining defects after design inspections.
However, a practitioner might want to know whether adopting a specific analysis
technique to predict remaining defects is more effective than expert opinion at
identifying design documents that require re-inspection.
Will lead either to changes in current software engineering practice or to
increased confidence in the value of current practice. For example, researchers
and practitioners would like to know under what conditions a project can safely
adopt agile technologies and under what conditions it should not.
Will identify discrepancies between commonly held beliefs and reality.

Nonetheless, there are systematic reviews that ask questions that are primarily of
interest to researchers. Such reviews ask questions that identify and/or scope future
research activities. For example, a systematic review in a PhD thesis should identify
Source: „Guidelines for performing Systematic Literature Reviews in SE“, Kitchenham et al., 2007
Examples

Kitchenham et al. [21] had three research questions:

Examples of research questions
Question 1: What evidence is there that cross-company estimation models are not
significantly different from within-company estimation models for predicting effort for
software/Web projects?
Question 2: What characteristics of the study data sets and the data analysis methods used in
the study affect the outcome of within- and cross-company effort estimation accuracy
studies?
Question 3: Which experimental procedure is most appropriate for studies comparing within-
and cross-company estimation models?

Jørgensen [17] had two research questions:

1. Should we expect more accurate effort estimates when applying expert judgment or
models?
2. When should software development effort estimates be based on expert judgment,
when on models, and when on a combination of expert judgment and models?

In both cases, the authors were aware from previous research that results were mixed, so in
each case they added a question aimed at investigating the conditions under which different
results are obtained.

5.3.2 Question Structure

Medical guidelines recommend considering a question about the effectiveness of a
treatment from three viewpoints:
The population, i.e. the people affected by the intervention.
Source: „Guidelines for performing Systematic Literature Reviews in SE“, Kitchenham et al., 2007
 2. When should software development effort estimates be based on expert judgment,
when on models, and when on a combination of expert judgment and models?

Question structure
In both cases, the authors were aware from previous research that results were mixed, so in
each case they added a question aimed at investigating the conditions under which different
results are obtained.

5.3.2 Question Structure

Medical guidelines recommend considering a question about the effectiveness of a
treatment from three viewpoints:
The population, i.e. the people affected by the intervention.
The interventions, which are usually a comparison between two or more
alternative treatments.
The outcomes, i.e. the clinical and economic factors that will be used to compare
10
the interventions.

More recently Petticrew and Roberts suggest using the PICOC (Population,
Intervention, Comparison, Outcome, Context) criteria to frame research questions
[25]. These criteria extend the original medical guidelines with:
Comparison: I.e. what is the intervention being compared with
Context: i.e. what is the context in which the intervention is delivered.

In addition, study designs appropriate to answering the review questions may be

identified and used to guide the selection of primary studies.
Source: „Guidelines for performing Systematic Literature Reviews in SE“, Kitchenham et al., 2007
Context: i.e. what is the context in which the intervention is delivered.

Question structure in SE
In addition, study designs appropriate to answering the review questions may be
identified and used to guide the selection of primary studies.

We discuss these criteria from the viewpoint of software engineering below.

Population
In software engineering experiments, the populations might be any of the following:
A specific software engineering role e.g. testers, managers.
A category of software engineer, e.g. a novice or experienced engineer.
An application area e.g. IT systems, command and control systems.
An industry group such as Telecommunications companies, or Small IT
companies.

A question may refer to very specific population groups e.g. novice testers, or
experienced software architects working on IT systems. In medicine the populations
are defined in order to reduce the number of prospective primary studies. In software
engineering far fewer primary studies are undertaken, thus, we may need to avoid any
restriction on the population until we come to consider the practical implications of
the systematic review.

Intervention
The intervention is the software methodology/tool/technology/procedure that
Source: „Guidelines for performing Systematic Literature Reviews in SE“, Kitchenham et al., 2007
are defined in order to reduce the number of prospective primary studies. In software
engineering far fewer primary studies are undertaken, thus, we may need to avoid any
restriction on the population until we come to consider the practical implications of
the systematic review.

Intervention
The intervention is the software methodology/tool/technology/procedure that
addresses a specific issue, for example, technologies to perform specific tasks such as
requirements specification, system testing, or software cost estimation.

Comparison
This is the software engineering methodology/tool/technology/procedure with which
the intervention is being compared. When the comparison technology is the
conventional or commonly-used technology, it is often referred to as the “control”
treatment. The control situation must be adequately described. In particular “not using
the intervention” is inadequate as a description of the control treatment. Software
engineering techniques usually require training. If you compare people using a
technique with people not using a technique, the effect of the technique is confounded
with the effect of training. That is, any effect might be due to providing training not
the specific technique. This is a particular problem if the participants are students.

Outcomes
Outcomes should relate to factors of importance to practitioners such as improved
reliability, reduced production costs, and reduced time to market. All relevant
Source: „Guidelines for performing Systematic Literature Reviews in SE“, Kitchenham et al., 2007
the intervention” is inadequate as a description of the control treatment. Software
engineering techniques usually require training. If you compare people using a
technique with people not using a technique, the effect of the technique is confounded
with the effect of training. That is, any effect might be due to providing training not
the specific technique. This is a particular problem if the participants are students.

Outcomes
Outcomes should relate to factors of importance to practitioners such as improved
reliability, reduced production costs, and reduced time to market. All relevant
outcomes should be specified. For example, in some cases we require interventions
that improve some aspect of software production without affecting another e.g.
improved reliability with no increase in cost. 11

A particular problem for software engineering experiments is the widespread use of

surrogate measures for example, defects found during system testing as a surrogate
for quality, or coupling measures for design quality. Studies that use surrogate
measures may be misleading and conclusions based on such studies may be less
robust.

Context
For Software Engineering, this is the context in which the comparison takes place
(e.g. academia or industry), the participants taking part in the study (e.g. practitioners,
academics, consultants, students), and the tasks being performed (e.g. small scale,
large scale). Many software
Source: „Guidelines experiments
for performing take place
Systematic Literature Reviews in academia
in SE“, Kitchenham etusing
al., 2007 student
measures may be misleading and conclusions based on such studies may be less
robust.

Context
For Software Engineering, this is the context in which the comparison takes place
(e.g. academia or industry), the participants taking part in the study (e.g. practitioners,
academics, consultants, students), and the tasks being performed (e.g. small scale,
large scale). Many software experiments take place in academia using student
participants and small scale tasks. Such experiments are unlikely to be representative
of what might occur with practitioners working in industry. Some systematic reviews
might choose to exclude such experiments although in software engineering, these
may be the only type of studies available.

Experimental designs
In medical studies, researchers may be able to restrict systematic reviews to primary
studies of one particular type. For example, Cochrane reviews are usually restricted to
randomised controlled trials (RCTs). In other circumstances, the nature of the
question and the central issue being addressed may suggest that certain study designs
are more appropriate than others. However, this approach can only be taken in a
discipline where the large number of research papers is a major problem. In software
engineering, the paucity of primary studies is more likely to be the problem for
systematic reviews and we are more likely to need protocols for aggregating
information from studies of widely different types.

Examples
Source: „Guidelines for performing Systematic Literature Reviews in SE“, Kitchenham et al., 2007
 The review protocol
Background. The rationale for the survey.
The research questions that the review is intended to answer.
The strategy that will be used to search for primary studies including search terms
and resources to be searched. Resources include digital libraries, specific journals,
and conference proceedings. An initial mapping study can help determine an
appropriate strategy.
Study selection criteria. Study selection criteria are used to determine which
studies are included in, or excluded from, a systematic review. It is usually
helpful to pilot the selection criteria on a subset of primary studies.
Study selection procedures. The protocol should describe how the selection
criteria will be applied e.g. how many assessors will evaluate each prospective
primary study, and how disagreements among assessors will be resolved.
Study quality assessment checklists and procedures. The researchers should
develop quality checklists to assess the individual studies. The purpose of the
quality assessment will guide the development of checklists.
Data extraction strategy. This defines how the information required from each
primary study will be obtained. If the data require manipulation or assumptions
and inferences to be made, the protocol should specify an appropriate validation
process.
Synthesis of the extracted data. This defines the synthesis strategy. This should
clarify whether or not a formal meta-analysis is intended and if so what
techniques will be used.
Dissemination strategy (if not already included in a commissioning document).
Project timetable. This should define the review schedule.
Source: „Guidelines for performing Systematic Literature Reviews in SE“, Kitchenham et al., 2007
previously,
Once researchers
the protocol should
has been expect
agreed, thetoreview
try outproper
each of thestart.
can stepsHowever,
describedasinnoted
this
section when
previously, they construct
researchers their
should research
expect to tryprotocol.
out each of the steps described in this

6.1
Identification of research
section when they construct their research protocol.
Identification of Research
6.1 Identification of Research
The aim of a systematic review is to find as many primary studies relating to the
research
The question
aim of as possible
a systematic reviewusing
is toan unbiased
find as manysearch strategy.
primary The
studies rigourtoofthe
relating the
search process
research is one
question factor that
as possible distinguishes
using an unbiasedsystematic reviewsThe
search strategy. from traditional
rigour of the
reviews.
search process is one factor that distinguishes systematic reviews from traditional
reviews.
6.1.1 Generating a search strategy
6.1.1 Generating
It is necessary a search
to determine strategy
and follow a search strategy. This should be developed in
consultation
It is necessarywith librarians and
to determine or others
followwith relevant
a search experience.
strategy. Searchbestrategies
This should developedarein
usually iterative
consultation withand benefitor
librarians from:
others with relevant experience. Search strategies are
Preliminary
usually searches
iterative and aimed
benefit at both identifying existing systematic reviews and
from:
assessing thesearches
Preliminary volume aimed
of potentially relevant studies.
at both identifying existing systematic reviews and
Trial searches
assessing using various
the volume combinations
of potentially relevantofstudies.
search terms derived from the
research
Trial question.
searches using various combinations of search terms derived from the
Checkingquestion.
research trial research strings against lists of already known primary studies.
Consultations
Checking trial with experts
research in the
strings field.lists of already known primary studies.
against
Consultations with experts in the field.
A general approach is to break down the question into individual facets i.e.
population,
A intervention,
general approach comparison,
is to break down the outcomes,
questioncontext, study designs
into individual as discussed
facets i.e.
in Section 5.3.2.
population, Then draw
intervention, up a list ofoutcomes,
comparison, synonyms, abbreviations,
context, and alternative
study designs as discussed
spellings.
in Section Other
5.3.2. terms can beupobtained
Then draw a list of by considering
synonyms, subject headings
abbreviations, used in
and alternative
Source: „Guidelines for performing Systematic Literature Reviews in SE“, Kitchenham et al., 2007
Jørgensen [16] investigated when we can expect expert estimates to have acceptable
However, publication
(e.g. time estimation
where failure
engineering
Publication bias
accuracy in comparison with formal models by reviewing relevant human judgement studies
bias remains
studies) a problem
and comparing their particularly forresults
results with the formalofexperiments,
software
to reject the null hypothesis is considered less interesting than an
studies.
experiment that is able to reject the null hypothesis. Publication bias is even more of a
problem when methods/techniques
6.1.2 Publication Bias are sponsored by influential groups in the software
industry.
PublicationForbias
example,
refers the
to theUSproblem
MoD is that
an extremely important
positive results and influential
are more likely to be
organisation
published than which sponsored
negative theThe
results. development
concept of of the Capability
positive or negativeMaturity
resultsModel and
sometimes
used its influence
depends to encourage
on the viewpoint of theindustry to adopt
researcher. (For the CMM.evidence
example, In such circumstances
that full
few companieswere
mastectomies wouldnotwant
alwaysto publish
requirednegative results
for breast andwas
cancer there is a strong
actually incentive
an extremely
to publishresult
positive papers
forthat support
breast thesufferers.)
cancer new method/technique.

Publication bias can lead

However, publication biastoremains
systematic bias in particularly
a problem systematic reviews unless
for formal special
experiments,
efforts are made
where failure to address
to reject thishypothesis
the null problem. Many of the standard
is considered search strategies
less interesting than an
identified
experiment above areable
that is usedto to address
reject this hypothesis.
the null issue including:
Publication bias is even more of a
Scanning
problem whenthe grey literature
methods/techniques are sponsored by influential groups in the software
Scanning
industry. conference
For example, theproceedings
US MoD is an extremely important and influential
organisation
Contacting which sponsored
experts the development
and researchers workingof inthe
theCapability Maturity
area and asking themModel and
if they
usedknow
its influence to encourageresults.
of any unpublished industry to adopt the CMM. In such circumstances
few companies would want to publish negative results and there is a strong incentive
to addition,
In publish papers that analysis
statistical support the new method/technique.
techniques can be used to identify the potential 15
significance of publication bias (see Section 6.5.7).
Publication bias can lead to systematic bias in systematic reviews unless special
6.1.3 Bibliography
efforts are Management
made to address and Document
this problem. Many of theRetrieval
standard search strategies
identified above are used to address this issue including:
Bibliographic
Scanning thepackages such as Reference Manager or Endnote may be useful for
grey literature
Source: „Guidelines for performing Systematic Literature Reviews in SE“, Kitchenham et al., 2007
6.1.3 Bibliography Management and Document Retrieval

Bibliographic packages such as Reference Manager or Endnote may be useful for

managing the large number of references that can be obtained from a thorough
Documenting the search
literature search.

Once reference lists have been finalised the full articles of potentially useful studies
will need to be obtained. A logging system is needed to make sure all relevant studies
are obtained.

6.1.4 Documenting the Search

The process of performing a systematic literature review must be transparent and
replicable (as far as possible):
The review must be documented in sufficient detail for readers to be able to
assess the thoroughness of the search.
The search should be documented as it occurs and changes noted and justified.
The unfiltered search results should be saved and retained for possible reanalysis.

Procedures for documenting the search process are given in Table 2.

Table 2 Search process documentation
Data Source Documentation
Digital Library Name of database
Search strategy for the database
Date
Source: „Guidelines of search
for performing Systematic Literature Reviews in SE“, Kitchenham et al., 2007
replicable (as far as possible):
The review must be documented in sufficient detail for readers to be able to
assess the thoroughness of the search.
The search should be documented as it occurs and changes noted and justified.

Documenting the search

The unfiltered search results should be saved and retained for possible reanalysis.

Procedures for documenting the search process are given in Table 2.

Table 2 Search process documentation
Data Source Documentation
Digital Library Name of database
Search strategy for the database
Date of search
Years covered by search
Journal Hand Searches Name of journal
Years searched
Any issues not searched
Conference proceedings Title of proceedings
Name of conference (if different)
Title translation (if necessary)
Journal name (if published as part of a journal)
Efforts to identify Research groups and researchers contacted (Names and contact details)
unpublished studies Research web sites searched (Date and URL)
Other sources Date Searched/Contacted
URL
Any specific conditions pertaining to the search

Researchers should specify their rationale for:

Source: „Guidelines for performing Systematic Literature Reviews in SE“, Kitchenham et al., 2007
that the number of papers addressing the topic would be small. Thus, both studies tried hard
to undertake a comprehensive search.

6.2
Study selection criteria
Study Selection

Once the potentially relevant primary studies have been obtained, they need to be
Inclusion andtheir
assessed for exclusion
actual criteria
relevance.should be based on the research question. They
should be piloted to ensure that they can be reliably interpreted and that they classify
6.2.1 correctly.
studies Study selection criteria
Study selection criteria are intended to identify those primary studies that provide
Examples
direct evidence about the research question. In order to reduce the likelihood of bias,
Kitchenham et al. used
selection criteria thebe
should following
decidedinclusion
duringcriteria:
the protocol definition, although they may
any study that compared predictions of cross-company models with within-
be refinedcompany
during the search process.
models based on analysis of single company project data.
Inclusion
They used and exclusion
the following criteriacriteria:
exclusion should be based on the research question. They
should bestudies
pilotedwhere
to ensure thatwere
projects theyonly
cancollected
be reliably
frominterpreted and that
a small number they classify
of different sources
(e.g. 2 or 3 companies),
studies correctly.
studies where models derived from a within-company data set were compared with
Examplespredictions from a general cost estimation model.
18
Kitchenham et al. used the following inclusion criteria:
Jørgensenany[17]study
included
that papers
comparedthatpredictions
compare judgment-based
of cross-company andmodels
model-based software
with within-
development effort estimation.
company He also
models based excluded
on analysis ofone relevant
single paper
company due to
project “incomplete
data.
information
They used theabout how the
following estimates
exclusion were derived”.
criteria:
studies where projects were only collected from a small number of different sources
Issues: (e.g. 2 or 3 companies),
Medical standards
studies where make
modelsa derived
point that it aiswithin-company
from important to avoid, as were
data set far ascompared
possible,with
predictions
exclusions basedfrom a general
on the costof
language estimation
the primarymodel.
study. This may not be so
important for Software Engineering.
Jørgensen [17] included papers that compare judgment-based and model-based software
It is possible
development effort
Source:
that inclusion
estimation.
„Guidelines He decisions
also
for performing excluded
Systematic
could one
Literature
be affected
relevant
Reviews
by knowledge
paper
in SE“, due to
Kitchenham
of the
“incomplete
et al., 2007
 obtained. However, Brereton et al. [5] point out that “
engineering abstracts is too poor to rely on when sele
More study selection criteria
should also review the conclusions.”

The next step is to apply inclusion/exclusion criteria

such as:
Language
Journal
Authors
Setting
Participants or subjects
Research Design
Sampling method
Date of publication.

Staples and Niazi point out that it is sometimes neces

that are not being addressed in order to refine your ex
Source: „Guidelines for performing Systematic Literature Reviews in SE“, Kitchenham et al., 2007
excluded papers with their advisor, an expert panel or other researchers. Alternatively,
individual researchers can apply a test-retest approach, and re-evaluate a random
sample of the primary studies found after initial screening to check the consistency of
Study quality assessment
their inclusion/exclusion decisions.

6.3 Study Quality Assessment

In addition to general inclusion/exclusion criteria, it is considered critical to assess the

“quality” of primary studies:
To provide still more detailed inclusion/exclusion criteria.
To investigate whether quality differences provide an explanation for differences
in study results.
As a means of weighting the importance of individual studies when results are
being synthesised.
To guide the interpretation of findings and determine the strength of inferences.
To guide recommendations for further research. 20

An initial difficulty is that there is no agreed definition of study “quality”. However,

the CRD Guidelines [19] and the Cochrane Reviewers’ Handbook [7] both suggest
that quality relates to the extent to which the study minimises bias and maximises
internal and external validity (see Table 3).

Table 3 Quality concept definitions

Term Synonyms Definition
Bias Systematic
Source: „Guidelines error Systematic
for performing A tendency toReviews
Literature produce results
in SE“, that depart
Kitchenham systematically
et al., 2007
 To guide the interpretation of findings and determine the strength of inferences.
To guide recommendations for further research.

Quality concept definitions

An initial difficulty is that there is no agreed definition of study “quality”. However,
the CRD Guidelines [19] and the Cochrane Reviewers’ Handbook [7] both suggest
that quality relates to the extent to which the study minimises bias and maximises
internal and external validity (see Table 3).

Table 3 Quality concept definitions

Term Synonyms Definition
Bias Systematic error A tendency to produce results that depart systematically
from the ‘true’ results. Unbiased results are internally valid
Internal validity Validity The extent to which the design and conduct of the study are
likely to prevent systematic error. Internal validity is a
prerequisite for external validity.
External validity Generalisability, The extent to which the effects observed in the study are
Applicability applicable outside of the study.

Most quality checklists (see Section 6.3.2) include questions aimed at assessing the
extent to which articles have addressed bias and validity.

6.3.1 The Hierarchy of Evidence

Medical guidelines suggest that an initial quality evaluation can be based on the type
of experiment design being used. Thus, we might rate a randomised controlled trial as
more trustworthy than an observational study. This has led to the concept of a
Source: „Guidelines for performing Systematic Literature Reviews in SE“, Kitchenham et al., 2007
quality 5).
Table of the
We study
do which
not were that
suggest scored to provide
anyone uses an
alloverall
the measure Researchers
questions. of study quality:
should
during
for
1. Iseach the trials
item
the data of
since
analysis the study
sometimes protocol
a simple
process appropriate? before
Yes/No being applied
answer mayto all
be the selected
misleading. studies.
Whatever
adopt Fink’s suggestion [11] which is to review the list of questions in the context of
form 1.1theWas the data
quality investigated
instrument to identify
takes, it shouldoutliers
be and to assess
assessed for distributional
reliability and properties
usability
their own study and select those quality evaluation questions that are most appropriate
Examples
for their
KitchenhamExample of quality assessment
before
during the
1.2 Was
analysis?
trials
specific
et
of the
theal.result
[21]
study
research protocol
of thequestions.
constructed
before
They
a quality
investigation may being
needapplied
questionnaire
used tobased
appropriately
to all the
construct selected
a measurement
on 5 issues
to transform
studies.
theaffecting
data andthe scale
select
for each
quality of item since
the study
appropriate sometimes
datawhich
points? a simple
were scored Yes/No
to provide an answer may be misleading.
overall measure Whatever
of study quality:
Examples
1.
2. Is the
Did data
studies analysis
carry outprocess appropriate?
a sensitivity oritresidual analysis?
form the
Kitchenham
quality
et al.
instrument
[21]
takes,
constructed a
should
quality
be assessed
questionnaire
for reliability
based on 5 issues
and usability
affecting the
1.1 Were
2.1 Was the thedata investigated
resulting to identify
estimation modelsoutliers
subjectand to assessordistributional
to sensitivity properties
residual analysis?
during
quality thethe
of
before
trials
study
analysis?
ofwhich
the study
were protocol
scored to before
providebeing applied
an overall to allofthe
measure selected
study studies.
quality:
2.2 Was the result of the sensitivity or residual analysis used to remove abnormal data
1. Ispoints
theWas
1.2 data analysis
the result process
if necessary? appropriate?
of the investigation used appropriately to transform the data and select
Examples
1.1 Was
appropriate
3. Were thedata
accuracy data investigated
points?
statistics based on to identify
the raw outliers
data scale? and to assess distributional properties
Kitchenham
before
2. How
4. Did studies
good etwas
al. the
analysis?
carry [21]
outconstructed
a sensitivity
study comparison aor
quality questionnaire
residual
method? analysis? based on 5 issues affecting the
quality
1.2of
2.1
4.1 the the
Was
Were
Was study
the
thesinglewhich
result of
resulting were
the scored
investigation
estimation
company selected to provide
models used
at an overall
appropriately
subject
random measure
to transform
to sensitivity
(not selected or of study
forresidual quality:
theanalysis?
convenience) data and
fromselect
1. Isseveral
theWas
2.2 datadifferent
appropriate analysis
the data
result process
points? appropriate?
of the sensitivity
companies? or residual analysis used to remove abnormal data
1.1
2. Did Was
studies
points
4.2 Was the data
carry
if necessary?
the investigated
out
comparison basedtoon
a sensitivity identify
or residual outliers
analysis?
an independent and to assess
hold out sample distributional properties
(0.5) or random
before
2.1 Were
3. Were
subsets analysis?
accuracythe resulting
(0.33), statistics estimation
based (0.17),
leave-one-out on themodels
raw
no holdsubject
data scale?
out to sensitivity
(0)? The scores or residual
used foranalysis?
this item reflect
1.2
4. How
the Was
2.2 good the
wasresult
researchers of thecomparison
theopinion
study investigation
sensitivitythe
regarding or used appropriately
residual
method?
stringency analysis
of eachused to transform
to removethe
criterion. data anddata
abnormal select
appropriate
points
4.1
5. The Was of data
sizeif necessary?
thethe
single points?
company selected
within-company data at set,random
measured (not selected
according fortoconvenience) from
the criteria presented
2.
3. Did
Were studies
accuracy
several
below. Whenever carrystatistics
different aout a sensitivity
companies?
study basedmore
used onortheresidual
thanrawone analysis?
data scale?
within-company data set, the average score
was 2.1
4. How
4.2 Were
good
Was the
used: thecomparison
was resulting
the studyestimation
based onmodels
comparison anmethod? subject tohold
independent sensitivity
out sample or residual
(0.5) or analysis?
random
2.2
4.1 Was
subsets
Less than the
10result
(0.33),single of the
company
leave-one-out
projects: Poorsensitivity
(0.17),or
selected
quality residual
at hold
no
(score random
= 0) analysis
out (0)? The used
(not selected tofor
scores remove abnormal
convenience)
used for data
from
this item reflect
points
several
the
Between if different
necessary?
researchers companies?
10 andopinion regarding
20 projects: the stringency
Fair quality (score =of each criterion.
0.33)
3.
5. Were
The accuracy
4.2 size
Was thethe
of statistics based
comparison basedonon theanraw data
independent scale? hold out sample (0.5)criteria
or random
Between 21 andwithin-company
40 projects: Good data set,
quality measured
(score according
= 0.67) to the presented
4. How
below. good(0.33),
subsets
Whenever was the studyused
study comparison
aleave-one-out (0.17),
more method?
thanno one
holdwithin-company
out (0)? The scores datausedset, forthethis item reflect
average score
More than 40 projects: Excellent quality (score = 1)
was4.1
the Was the single
researchers
used: company
opinion selected
regarding the at random (not
stringency selected
of each for convenience) from
criterion.
several
5. The
Lesssize
thandifferent
of10the companies?
within-company
projects: Poor quality data set, =
(score measured
0) according to the criteria presented
4.2
below. Was the
Whenever
Between comparison
10 and a study based
used
20 projects: more
Faironquality
an
thanindependent
one
(score hold out sample
within-company
= 0.33) (0.5)the
data set, or average
random score
wassubsets
used: (0.33),
Between 21 andleave-one-out
40 projects: Good (0.17),quality
no hold out (0)?
(score The scores used for this item reflect
= 0.67)
the
Lessresearchers
than opinion regarding the stringency of each criterion.
More than 10 40 projects:
projects: Poor quality
Excellent (score
quality = 0)= 1)
(score 23
5. The size of
Between 10 the
andwithin-company
Source: 20 projects:
„Guidelines Fairdata
for performing set,(score
quality
Systematic measured
Literature =Reviews
0.33)according to etthe
in SE“, Kitchenham criteria presented
al., 2007
The objective of this stage is to design data extraction forms to accurately record the
Data extraction forms
information researchers obtain from the primary studies. To reduce the opportunity
for bias, data extraction forms should be defined and piloted when the study protocol
is defined.

6.4.1 Design of Data Extraction Forms

The data extraction forms must be designed to collect all the information needed to
address the review questions and the study quality criteria. If the quality criteria are to
be used to identify inclusion/exclusion criteria, they require separate forms (since the
information must be collected prior to the main data extraction exercise). If the quality
criteria are to be used as part of the data analysis, the quality criteria and the review
data can be included in the same form.

In most cases, data extraction will define a set of numerical values that should be
extracted for each study (e.g. number of subjects, treatment effect, confidence
intervals, etc.). Numerical data are important for any attempt to summarise the results
of a set of primary studies and are a prerequisite for meta-analysis (i.e. statistical
techniques aimed at integrating the results of the primary studies).
29
Data extraction forms need to be piloted on a sample of primary studies. If several
researchers will use the forms, they should all take part in the pilot. The pilot studies
are intended to assess both technical issues such as the completeness of the forms and
usability issues such as the clarity of user instructions and the ordering of questions.
Source: „Guidelines for performing Systematic Literature Reviews in SE“, Kitchenham et al., 2007
 Examples
Kitchenham et al. [21] used the extraction form shown in Table 7 (note the actual form also
included the quality questions).

Example of a data extraction form

Table 7 Data Collection form completed for Maxwell et al., 1998
Data item Value Additional notes
Data Extractor
Data Checker
Study Identifier S1
Application domain Space, military and industrial
Name of database European Space Agency (ESA)
Number of projects in 108
database (including within-
company projects)
Number of cross-company 60
projects
Number of projects in within- 29
company data set
Size metric(s): FP: No
FP (Yes/No) LOC: Yes (KLOC)
Version used: Others: No
LOC (Yes/No)
Version used:
Others (Yes/No)
Number:
Number of companies 37
Number of countries 8 European only
represented
Were quality controls applied No
to data collection?
If quality control, please
describe
How was accuracy Measures:
2
measured? R (for model construction only)
MMRE
Pred(25)
r (Correlation between estimate
and actual)
Source: „Guidelines for performing Systematic Literature Reviews in SE“, Kitchenham et al., 2007
Cross-company model
What technique(s) was used A preliminary productivity analysis
to construct the cross- was used to identify factors for
company model? inclusion in the effort estimation
model.
Generalised linear models (using
SAS). Multiplicative and Additive
models were investigated. The
multiplicative model is a
logarithmic model.
If several techniques were In all cases, accuracy assessment It can be assumed that
used which was most was based on the logarithmic linear models did not work
accurate? models not the additive models. well.
What transformations if any Not clear whether the variables Not important: the log
were used? were transformed or the GLM was models were used and they
used to construct a log-linear were presented in the raw
model data form – thus any
accuracy metrics were
based on raw data
predictions.
What variables were KLOC, Language subset, Category Category is the type of
included in the cross- subset, RELY application.
company model? RELY is reliability as
defined by Boehm (1981)
What cross-validation A hold-out sample of 9 projects
method was used? from the single company was used
to assess estimate accuracy
Was the cross-company Yes The baseline was the
model compared to a correlation between the
baseline to check if it was estimates and the actuals
better than chance? for the hold-out.
What was/were the The correlation between the
measure(s) used as prediction and the actual for the
benchmark? single company was tested for
statistical significance. (Note it was
significantly different from zero for
the 20 project data set, but not the
9 project hold-out data set.)
Source: „Guidelines for performing Systematic Literature Reviews in SE“, Kitchenham et al., 2007
benchmark? single company was tested for
statistical significance. (Note it was
significantly different from zero for
the 20 project data set, but not the
9 project hold-out data set.)

Within-company model
What technique(s) was used A preliminary productivity analysis
to construct the within- was used to identify factors for
company model? inclusion in the effort estimation
model.

Generalised linear models (using

SAS). Multiplicative and Additive
models were investigated. The
multiplicative model is a
logarithmic model.
If several techniques were In all cases, accuracy assessment It can be assumed that
used which was most was based on the logarithmic linear models did not work
accurate? models not the additive models. well.
What transformations if any Not clear whether the variables Not important: the log
were used? were transformed or the GLM was models were used and they
used to construct a log-linear were presented in the raw
model data form – thus any
accuracy metrics were
based on raw data
predictions.
What variables were KLOC, Language subset, Year
included in the within-
company model?
What cross-validation A hold-out sample of 9 projects
method was used from the single company was used
to assess estimate accuracy
Comparison
What was the accuracy Accuracy on main single company Using the 79 cross- 31
obtained using the cross- data set (log model): company projects, Maxwell
company model? n=11 (9 projects omitted) et al. identified the best
MMRE=50% model for that dataset and
Pred(25)=27%
Source: „Guidelines for performing theKitchenham
Systematic Literature Reviews in SE“, best modeletforal.,the
2007
 r, correlation between the
What statistical tests were prediction and the actual
used to compare the results?

What were the results of the

tests?

Data Summary
Data base summary (all Effort min: 7.8 MM KLOC: non-blank, non-
projects) for size and effort Effort max: 4361 MM comment delivered 1000
metrics. Effort mean: 284 MM lines. For reused code
Effort median: 93 MM Boehm’s adjustment were
Size min: 2000 KLOC made (Boehm, 1981).
Size max: 413000 KLOC Effort was measured in
Size mean: 51010 KLOC man months, with 144 man
Size median: 22300 KLOC hours per man month
With-company data Effort min: Not specified
summary for size and effort Effort max:
metrics. Effort mean:
Effort median:
Size min:
Size max:
Size mean:
Size median:

Jørgensen [17] extracted design factors and primary study results. Design factors included:
Study design
Estimation method selection process
Estimation models
Source: „Guidelines for performing Systematic Literature Reviews in SE“, Kitchenham et al., 2007
 Accuracy
Variance
Other results
Data extraction procedures
Jørgensen’s article includes the completed extraction form for each primary study.

6.4.3 Data extraction procedures

Whenever feasible, data extraction should be performed independently by two or
more researchers. Data from the researchers must be compared and disagreements
resolved either by consensus among researchers or arbitration by an additional
independent researcher. Uncertainties about any primary sources for which agreement
cannot be reached should be investigated as part of any sensitivity analyses. A
separate form must be used to mark and correct errors or disagreements.

If several researchers each review different primary studies because time or resource
constraints prevent all primary papers being assessed by at least two researchers, it is
important to employ some method of checking that researchers extract data in a
consistent manner. For example, some papers should be reviewed by all researchers
(e.g. a random sample of primary studies), so that inter-researcher consistency can be
assessed.

For single researchers such as PhD students, other checking techniques must be used.
For example supervisors could perform data extraction on a random sample of the
primary studies and their results cross-checked with those of the student.
Alternatively, a test-retest process can be used where the researcher performs a
second extraction from a random selection of primary studies to check data extraction
consistency.

ExamplesSource: „Guidelines for performing Systematic Literature Reviews in SE“, Kitchenham et al., 2007
meta-analysis see [7].)

Descriptive (narrative) data synthesis

The data synthesis activities should be specified in the review protocol. However,
some issues cannot be resolved until the data is actually analysed, for example, subset
analysis to investigate heterogeneity is not required if the results show no evidence of
heterogeneity.

6.5.1 Descriptive (Narrative) synthesis

Extracted information about the studies (i.e. intervention, population, context, sample
sizes, outcomes, study quality) should be tabulated in a manner consistent with the
review question. Tables should be structured to highlight similarities and differences
between study outcomes.

It is important to identify whether results from studies are consistent with one another
(i.e. homogeneous) or inconsistent (e.g. heterogeneous). Results may be tabulated to
display the impact of potential sources of heterogeneity, e.g. study type, study quality,
and sample size.

34

Source: „Guidelines for performing Systematic Literature Reviews in SE“, Kitchenham et al., 2007
He concluded that models are not systematically better than experts for software cost
estimation, possibly because experts possess more information than models or it may be

Quantitative data synthesis

difficult to build accurate software development estimation models. Expert opinion is likely to
be useful if models are not calibrated to the company using them and/or experts have access
to important contextual information that they are able to exploit. Models (or a combination of
models and experts) may be useful when there are situational biases towards overoptimism,
experts do not have access to large amounts of contextual information, and/or models are
calibrated to the environment.

6.5.2 Quantitative Synthesis

Quantitative data should also be presented in tabular form including:
Sample size for each intervention.
Estimates effect size for each intervention with standard errors for each effect.
Difference between the mean values for each intervention, and the confidence
interval for the difference.
Units used for measuring the effect.

However, to synthesise quantitative results from different studies, study outcomes

must be presented in a comparable way. Medical guidelines suggest different effect
measures for different types of outcome.

Binary outcomes (Yes/No, Success/Failure) can be measured in several different

ways:
Odds. The ratio of the number of subjects in a group with an event to the number
Source: „Guidelines for performing Systematic Literature Reviews in SE“, Kitchenham et al., 2007
 To avoid the problems of post-hoc analysis (i.e. “fishing” for results), rese
should identify possible sources of heterogeneity when they construct the
Example
protocol.of
Forpresenting quantitative
example, studies of data
different types may have different results
often useful toas a forest
synthesise plotof different study types separately and
the results
whether the results are consistent across the different study types.

Figure 1 Example of a forest plot

Study 1

Study 2

Study 3

-0.2 -0.1 0 0.1 0.2

Favours control Favours intervention

6.5.4 Qualitative Synthesis

Source: „Guidelines for performing Systematic Literature Reviews in SE“, Kitchenham et al., 2007