Performing systematic literature review

in software engineering

Zlatko Stapić1, Eva García López2, Antonio García Cabot2,

Luis de Marcos Ortega2, Vjeran Strahonja1
1 University of Zagreb, Faculty of Organization and Informatics
Pavlinska 2, 42000 Varaždin, Croatia,
{zlatko.stapic, vjeran.strahonja}@foi.hr

2 University of Alcalá, Computer Science Department

Ctra. Barcelona km 33.6, 28871, Alcalá de Henares (Madrid), Spain
{eva.garcial, a.garciac, luis.demarcos}@uah.es

CECIIS 2012 - 20.09.2012. – Varaždin, Croatia

 Agenda

Introduction

The definition and history of SLR in SE

Performing SLR in SE - three phases

Light SLR

Discussion

Conclusion

2 Stapić, García L., García C., de Marcos O., Strahonja: Performing SLR in SE – CECIIS, 20th September 2012
 Introduction

• Systematic vs. unsystematic approach

• Systematic approach in literature review
 comprehensive and thorough analysis
 creation of solid theoretical and state-of-the-art basis for the rest of
the research
• Several possibilities in systematic approach [1], [2]
 systematic literature review (SLR)
 systematic mapping studies
 tertiary reviews
 narrative review
 conceptual review
 rapid review, etc.

3 Stapić, García L., García C., de Marcos O., Strahonja: Performing SLR in SE – CECIIS, 20th September 2012
 Introduction

• Reasons to use SLR

 to use the scientific approach
 to summarize the existing evidence on a treatment or a technology
 to identify any gaps in current research
 to provide a framework/background in order to appropriately position
new research activities
 to meet the quality expectations

• most important reason is [3]:

 to minimize the chances of making wrong conclusions which could be
the result of biases either in primary studies or in the review process
itself.

4 Stapić, García L., García C., de Marcos O., Strahonja: Performing SLR in SE – CECIIS, 20th September 2012
 The definition and history of SLR in SE

• Definition
 “A systematic literature review is a means of evaluating and
interpreting all available research relevant to a particular research
question, topic area, or phenomenon of interest. Systematic reviews
aim to present a fair evaluation of a research topic by using a
trustworthy, rigorous, and auditable methodology” [1].
• The origins [17],[5]
 The origins of SR are back to the beginning of the 20th century
 In 1980’s, systematic research synthesis and meta-analysis are widely
popular in the domain of health sciences
 Different synonyms of this method have been used in the literature
 research review,
 research synthesis,
 research integration,
 systematic overview et cetera.

5 Stapić, García L., García C., de Marcos O., Strahonja: Performing SLR in SE – CECIIS, 20th September 2012
 The definition and history of SLR in SE

• The history of SLR in SE

 2004: The guideline for SLR in SE was proposed by Kitchenham in [4].
 adaptation of several existing guidelines from other disciplines,
mainly medicine.
 three proposed phases of systematic review (planning the review,
conducting the review and reporting the review)
 some authors [5] [6] [7] found that Kitchenham described SLR
phases at a relatively high level and partially inappropriate to
conduct for researchers in the field of SE.
 2007: Kitchenham and Charters published a new version of
technology report [1]
 The basis for this guideline remained the same: existing guidelines
used by medical researchers, but was reinforced by several books
and discussions with researches from other fields.

6 Stapić, García L., García C., de Marcos O., Strahonja: Performing SLR in SE – CECIIS, 20th September 2012
 Performing SLR in SE - three phases

General steps of the methodology [1]:  Phase 1: Planning the review

 Identification of the need for a review
 Commissioning a review (optional)
 Specifying the research question(s)
 Developing a review protocol
 Evaluating the review protocol (recommended)
 Phase 2: Conducting the review
 Identification of research
 Selection of primary studies
 Study quality assessment
 Data extraction and monitoring
 Data synthesis
 Phase 3: Reporting the review
 Specifying dissemination mechanisms
 Formatting the main report
 Evaluating the report (recommended)

7 Stapić, García L., García C., de Marcos O., Strahonja: Performing SLR in SE – CECIIS, 20th September 2012
 Performing SLR in SE – Phase 1

• Phase 1: Planning the review

 The results of this phase should be a clearly defined review protocol
containing the purpose and the procedures of the review.

 Identification of the need for a review

 check and evaluate existing reviews on the same topic
 there are several checklists, such as one in [8] that could be used
for evaluation of existing
 Commissioning a review (optional)
 if performed, commissioning document should be provided by
stakeholders. This document should contain all important data
about required work.
 Specifying the research question(s)
 Developing a review protocol
 Evaluating the review protocol (recommended)

8 Stapić, García L., García C., de Marcos O., Strahonja: Performing SLR in SE – CECIIS, 20th September 2012
 Performing SLR in SE – Phase 1

• Phase 1: Planning the review

 …
 Specifying the research question(s)
 probably most important activity in whole SLR
 the questions will define which primary studies to include in SLR
 the questions should be answered at the end of SLR process
 Developing a review protocol
 also important activity in this phase
 detailed review protocol will determine the rest of the review
process
 the protocol will probably be the subject of constant changes
through the process [7], but it is neccessary to reduce the
possibility of bias
 some aspects of the review protocol should be piloted during the
protocol development process

9 Stapić, García L., García C., de Marcos O., Strahonja: Performing SLR in SE – CECIIS, 20th September 2012
 Performing SLR in SE – Phase 1

• Phase 1: Planning the review

 Protocol template proposed by Biolchini et al. in [5]

10 Stapić, García L., García C., de Marcos O., Strahonja: Performing SLR in SE – CECIIS, 20th September 2012
 Performing SLR in SE – Phase 1

• Phase 1: Planning the review

 …
 Evaluating the review protocol (recommended)
 several methods such as
 authors review
 peer review
 review by supervisor
 review by external experts
 test of protocol execution etc.
 evaluation is not obligatory but it is a means of improving the
overall quality of the review.

11 Stapić, García L., García C., de Marcos O., Strahonja: Performing SLR in SE – CECIIS, 20th September 2012
 Performing SLR in SE – Phase 2

• Phase 2: Conducting the review

 Identification of research
 creation of a list of entire population of publications relevant to
the research questions and obtained by performing a search
strategy.
 the obtained list of documents should be time stamped and the
references to the documents should be kept.
 Relevant sources
 Brereton et al. [9] identified seven electronic sources as most
relevant sources to SErs (IEEExplore, ACM Digital Library, Google
Scholar, Citeseer library, INSPEC, ScienceDirect and EI Compendex),
and they also discuss about considering the use of additional sources
(SpringerLink, Web of Science and SCOPUS) from publishers or
bibliographical databases.
 Other authors, e.g. [10] and [1] also tried to identify a list of relevant
journals and conferences in the field of software engineering and
obtained a list of relevant journals and conferences.

12 Stapić, García L., García C., de Marcos O., Strahonja: Performing SLR in SE – CECIIS, 20th September 2012
 Performing SLR in SE – Phase 2

• Phase 2: Conducting the review

 …
 Selection of primary studies
 Application of inclusion and exclusion criteria on initial list of
documents
 The inclusion criteria will define which of these studies to include in
the set of relevant ones,
 the exclusion criteria could be applied on the already selected studies
in order to identify those that do not meet additional conditions; or
on the initial list of studies in order to remove irrelevant ones.
 the inclusion and exclusion are based on research questions, study
types etc.

13 Stapić, García L., García C., de Marcos O., Strahonja: Performing SLR in SE – CECIIS, 20th September 2012
 Performing SLR in SE – Phase 2

• Phase 2: Conducting the review

 Selection of primary studies – an example process [3]
•

14 Stapić, García L., García C., de Marcos O., Strahonja: Performing SLR in SE – CECIIS, 20th September 2012
 Performing SLR in SE – Phase 2

• Phase 2: Conducting the review

 …
 Study quality assessment
 making sure that study findings are relevant and unbiased
 not an easy task - no definition of „study quality”
 Examples in [3], [11], [12], [13] and [14].
 The following elements should be assessed regardless of the
study type:
 Appropriateness of study design to the research objective.
 Risk of bias.
 Choice of outcome measure.
 Statistical issues.
 Quality of reporting and intervention.
 Generalizability.
 the most useful tool to use are checklists

15 Stapić, García L., García C., de Marcos O., Strahonja: Performing SLR in SE – CECIIS, 20th September 2012
 Performing SLR in SE – Phase 2

• Phase 2: Conducting the review

 …
 Data extraction and monitoring
 extracting relevant information from selected studies
 extraction forms are usually used
 creation of extraction forms is not an trivial task
 According to [1] and [15], data extraction form could include
 extraction information (data extractor, data checker and date of
extraction),
 general study information (study identifier, title and publication
details),
 questions to answer review questions,
 questions to assess study quality, and
 data summary.
 Examples in [14], [15] and [16].
 Data synthesis

16 Stapić, García L., García C., de Marcos O., Strahonja: Performing SLR in SE – CECIIS, 20th September 2012
 Performing SLR in SE – Phase 2

• Phase 2: Conducting the review

 …
 Data synthesis
 extracted data are collected and summarized
 two types of synthesis [8]
 descriptive (narrative)
 quantitative
 these approaches are not mutually exclusive [9]
 synthesis depends on review protocol but also on the type of
research questions
 Kitchenham suggests that researchers should “synthesize the
quantitative and qualitative studies separately, and then attempt
to integrate the results by investigating whether the qualitative
results can help explain the quantitative results”
 useful tools are tables, charts etc.

17 Stapić, García L., García C., de Marcos O., Strahonja: Performing SLR in SE – CECIIS, 20th September 2012
 Performing SLR in SE – Phase 3

• Phase 3: Reporting the review

 The aim of the final phase of the systematic literature review process
is to write the results of the review in a form suitable to dissemination
channel and target audience or parties.

 Specifying dissemination mechanisms

 defined during commissioning activities or in review protocol
 the results should be disseminated by appropriate means
 academic journals and conferences
 technical reports
 web pages
 posters
 practitioner oriented magazines etc.
 Formatting the main report
 Evaluating the report (recommended)

18 Stapić, García L., García C., de Marcos O., Strahonja: Performing SLR in SE – CECIIS, 20th September 2012
 Performing SLR in SE – Phase 3

• Phase 3: Reporting the review

 …
 Formatting the main report
 generally accepted report structure is given by CRD’s guidelines
from 2001, updated in 2009 [8], and in SE accepted by [1] and
others
 Evaluating the report (recommended)
 depends on the type of publication
 review by independent peer reviewers (for academic papers=
 review by supervisor (for doctoral thesis)
 review by the same expert panel that review the defined protocol (for
technical reports) etc.

19 Stapić, García L., García C., de Marcos O., Strahonja: Performing SLR in SE – CECIIS, 20th September 2012
 Light SLR

• The need for Light SLR

 SLR is exhausting and time consuming process
 SLR is defined to be performed by a group of researchers
 What about for example PhD students?!?

• The answer is Light SLR

• The authors of the guidelines conclude that Light version would be
sufficient for specific cases such as writing a dissertation.

20 Stapić, García L., García C., de Marcos O., Strahonja: Performing SLR in SE – CECIIS, 20th September 2012
 Light SLR

 The light process should include these steps

 Developing a protocol
 Defining the research question(s).
 Specifying what will be done to address the problem of a single
researcher applying inclusion/exclusion criteria and undertaking all
the data extraction.
 Defining the search strategy.
 Defining the data to be extracted from each primary study
including quality data.
 Maintaining lists of included and excluded studies.
 Using the data synthesis guidelines.
 Using the reporting guidelines.
 These steps should be performed rigorously and taking into
consideration all notes and recommendations given by the authors.

21 Stapić, García L., García C., de Marcos O., Strahonja: Performing SLR in SE – CECIIS, 20th September 2012
 Discussion

• Main advantages [1]:

 The methodology is well-defined
 it enables researchers to provide the information available in the wide
range of sources
 in the case of quantitative data, it is possible to perform some meta-
analysis and to extract information that single study cannot provide
 if compared to unstructured methods, SLR is more reliable and more
likely to be unbiased
• Main disadvantages:
 requires lots of time and efforts
 saturated by many review points
 search term pilot reviews, protocol reviews, initial selection
reviews, final selection reviews, data extraction reviews, and data
analysis reviews [7]
 the possibility of using meta-analysis could also be disadvantage [1]

22 Stapić, García L., García C., de Marcos O., Strahonja: Performing SLR in SE – CECIIS, 20th September 2012
 Discussion

• Conducting an SLR in SE is much harder than in other disciplines

[5]
• Some authors discuss that overall process is hard to conduct,
especially:
 protocol development
 studies search
 studies evaluation, etc.
• SLR also depends on solid literature coverage and can not be to
explore new or revolutionary phenomena
• The process is hardly documented ad review protocol is usually
changed during the SLR process.

23 Stapić, García L., García C., de Marcos O., Strahonja: Performing SLR in SE – CECIIS, 20th September 2012
 Conclusion

• The method overview presented in this paper is based on the

Kitchenham’s guidelines presented in [1] and is expanded by reported
feedback of the researchers, mainly from the field of software
engineering.

• Although not an easy, this method is useful and could be used to

decrease the biases and to increase the review quality.
• The usage in SE has serious obstacles and in order to overcome these
obstacles:
 scope of the review should be limited by choosing clear and narrow research
questions
 whole process should be in advance well defined by putting an considerable
efforts in creation of feasible review protocol
• SLR still emerges in the field of SE, so we encourage
 publishing the replications of existing systematic reviews
 creation of centralized index of existing literature reviews.

24 Stapić, García L., García C., de Marcos O., Strahonja: Performing SLR in SE – CECIIS, 20th September 2012
 Looking forward to your qeustions 

25 Stapić, García L., García C., de Marcos O., Strahonja: Performing SLR in SE – CECIIS, 20th September 2012
 References

[1] B. Kitchenham and S. Charters, “Guidelines for performing Systematic Literature reviews in
Software Engineering Version 2.3,” Keele University and University of Durham, Technical report
EBSE-2007-01, 2007.
[2] M. Petticrew and H. Roberts, Systematic reviews in the social sciences: a practical guide.
Malden, MA: Blackwell Pub., 2005.
[3] T. Dybå and T. Dingsøyr, “Strength of evidence in systematic reviews in software engineering,”
2008, pp. 178–187.
[4] B. Kitchenham, “Procedures for Performing Systematic Reviews,” Software Engineering Group;
National ICT Australia Ltd., Keele; Eversleigh, Technical report Keele University Technical
Report TR/SE-0401; NICTA Technical Report 0400011T.1, 2004.
[5] J. Biolchini, P. Gomes Mian, A. Candida Cruz Natali, and G. Horta Travassos, “Systematic
Review in Software Engineering,” PESC, Rio de Janeiro, Technical report RT - ES 679 / 05,
2005.
[6] P. Mian, T. Conte, A. Natali, J. Biolchini, and G. Travassos, “A Systematic Review Process for
Software Engineering,” in ESELAW ’05: 2nd Experimental Software Engineering Latin American
Workshop, 2005.
[7] M. Staples and M. Niazi, “Experiences using systematic review guidelines,” Journal of Systems
and Software, vol. 80, no. 9, pp. 1425–1437, 2007.
[8] Centre for Reviews and Dissemination, University of York, Systematic reviews: CRD’s guidance
for undertaking reviews in health care. York: Centre for Reviews and Dissemination, 2009.
[9] P. Brereton, B. A. Kitchenham, D. Budgen, M. Turner, and M. Khalil, “Lessons from applying the
systematic literature review process within the software engineering domain,” Journal of
Systems and Software, vol. 80, no. 4, pp. 571–583, 2007.
26 Stapić, García L., García C., de Marcos O., Strahonja: Performing SLR in SE – CECIIS, 20th September 2012
 References

[10] J. Hannay, D. Sjoberg, and T. Dyba, “A Systematic Review of Theory Use in Software
Engineering Experiments,” IEEE Transactions on Software Engineering, vol. 33, no. 2, pp. 87–
107, 2007.
[11] B. Kitchenham, O. Pearl Brereton, D. Budgen, M. Turner, J. Bailey, and S. Linkman,
“Systematic literature reviews in software engineering – A systematic literature review,”
Information and Software Technology, vol. 51, no. 1, pp. 7–15, 2009.
[12] B. Kitchenham, R. Pretorius, D. Budgen, O. Pearl Brereton, M. Turner, M. Niazi, and S.
Linkman, “Systematic literature reviews in software engineering - A tertiary study,”
Information and Software Technology, vol. 52, no. 8, pp. 792–805, 2010.
[13] B. Kitchenham, P. Brereton, M. Turner, M. Niazi, S. Linkman, R. Pretorius, and D. Budgen,
“The impact of limited search procedures for systematic literature reviews - A participant-
observer case study,” 2009, pp. 336–345.
[14] M. Unterkalmsteiner, T. Gorschek, A. K. M. M. Islam, C. K. Cheng, R. B. Permadi, and R. Feldt,
“Evaluation and Measurement of Software Process Improvement - A Systematic Literature
Review,” IEEE Transactions on Software Engineering, 2012.
[15] M. Jørgensen, “Estimation of Software Development Work Effort:Evidence on Expert
Judgment and Formal Models,” International Journal of Forecasting, vol. 23, no. 3, pp. 449–
462, 2007.
[16] T. Dybå and T. Dingsøyr, “Empirical studies of agile software development: A systematic
review,” Information and Software Technology, vol. 50, no. 9–10, pp. 833–859, 2008.
[17] B. J. Williams and J. C. Carver, “Characterizing software architecture changes: A systematic
review,” Information and Software Technology, vol. 52, no. 1, pp. 31–51, 2010.

27 Stapić, García L., García C., de Marcos O., Strahonja: Performing SLR in SE – CECIIS, 20th September 2012
 Acknowledgments

• The research presented in this paper was conducted as a part of a PhD

research and is partially founded by the scholarship granted to the first
author from Croatian Science Foundation.

28 Stapić, García L., García C., de Marcos O., Strahonja: Performing SLR in SE – CECIIS, 20th September 2012
 Abstract

• In the field of software engineering (SE) the first primary studies

about Systematic Literature Review (SLR) have been conducted
during the last decade. By adapting guidelines from other
disciplines Kitchenham in 2004 created guidelines on performing
the SLR in a field of SE. Since then the SLR is used in SE regularly,
but many obstacles still remain.
This paper aims to give an short but complete overview of all
phases and stages that should be undertaken in SLR, and although
based on mentioned guidelines, this paper takes into consideration
feedback and experiences reported by other authors and discusses
the possible approaches along with advantages and
dissadvantages of the method.

29 Stapić, García L., García C., de Marcos O., Strahonja: Performing SLR in SE – CECIIS, 20th September 2012