ISSN 2255-9094 (online)

Information Technology and Management Science ISSN 2255-9086 (print)

December 2017, vol. 20, pp. 48–53
doi: 10.1515/itms-2017-0008
https://www.degruyter.com/view/j/itms

Test Case Review Processes in Software Testing

Oksana Petunova1, Solvita Bērziša2
1, 2
Riga Technical University, Latvia

Abstract – Qualitative system requirements and thoughtful This immediately increases project costs. Fixing such defects is
communication are the key factors to successful implementation of more expensive and labour-intensive because it is necessary to
software development projects. However, errors that occur by
fix defects not only in a software code, but also in the
misunderstanding and incomprehension between parties involved
in the project can lead not only to the high cost of the project but documentation where this functionality is described [2], [5]. In
also to a large number of defects discovered only after the system order to decrease the number of defects detected at the end of
release, which in turn strongly influences the product quality. The testing or after the release of the software, it is necessary to take
software review processes are implemented to reduce projects preventive measures starting with the project earliest phases.
costs and ensure high product quality. The goal of the present One of these preventive measures is static testing [6], [10].
paper is to identify the role of review processes in software testing. Static testing provides a good opportunity to improve
To achieve this goal, the process of test case review has been
implemented during testing.
software quality and reliability. Its techniques provide a
possibility to get early feedback on software quality. The static
Keywords – Communication, inspection, software review process, testing techniques can be used without a computer because
software testing, static testing. software testing is performed without software executing. That
is why static testing is convenient to use at the project earliest
I. INTRODUCTION phases. One of the static testing techniques is software review
Successful use of any software depends on users’ satisfaction process [6].
and its usefulness. In order to ensure high quality and maximum The goal of the present paper is to identify the role of
efficiency of the daily users’ work, any software should operate software review processes in software testing. To achieve this
continuously and uninterrupted. goal, one of the software review types – a test case review – has
Software testing plays an important role in the software been introduced in software testing for the case study project.
development lifecycle [2]. Software testing is a process rather The case study has been conducted on the basis of the
than a single activity. This process starts with test planning, following tasks: to identify the role of communication and test
designing test cases, preparing for execution, evaluating a status case reviews in the software testing; to find out if timely
and ends with the test closure. During software testing, the information availability and the test case reviews can improve
developed software is tested for compliance with the software quality; to determine if the test case reviews can
international standards, requirements and business needs. If reduce total testing time.
discrepancy and shortcomings are found in software during the The paper is organised as follows: Section 2 describes
testing, it means there are defects and errors which should be theoretical foundation and literature review of related studies.
fixed. Section 3 describes case study process, a project used for the
According to the seven principles of testing, it is irrational case study and data collected for the analysis. Section 4
and impossible to check all combinations of software inputs and summarises the results of the test cases review implementation.
preconditions and find all defects in the software product [7]. Section 5 describes the implications of these results and the
The software testing can provide an ability to reduce the number conclusions that may be drawn.
of undetected defects in the developed software. Even if defects
are not found, testing cannot prove that software is 100 % defect II. RELATED RESEARCH
free [6]. In order to organise effective and useful software In general, software review process is one of the few methods
testing, it is necessary to choose appropriate testing techniques. that can be used for error detection and correction in the
Test cases should also be defined based on project and product software development process. The achievements of this
risks [1], [6]. process can be associated with a possibility to detect and fix
Many errors and defects are discovered at the end of the defects at the projects earliest phases when defect prevention
testing or are not discovered at all until users find them after the costs are low in comparison with the projects latest phases [6],
release of the software [2], [9], [12]. Low software quality can [8], [9], [11].
reduce product reputation and increase the possibility that end Software review process is one of the most effective and
users and customers prefer using competitors’ services [1], [2], productive methods how to assess and verify the quality of the
[5], [9]. software and its artefacts in the software development lifecycle.
Defects which are discovered at the end of testing and defects It is one of static testing (verification) techniques, and its main
which are found by software users are much more expensive to objective is to find and avoid an error which appears during the
fix than defects which are discovered at the earliest project software development process. Software review process can be
phases when, for example, business requirements were defined. applied to any artefact created during the software development

©2017 Oksana Petunova, Solvita Bērziša. 48

This is an open access article licensed under the Creative Commons Attribution License
(http://creativecommons.org/licenses/by/4.0), in the manner agreed with De Gruyter Open.

Unauthentifiziert | Heruntergeladen 29.01.20 13:42 UTC

 Information Technology and Management Science
_______________________________________________________________________________________________ 2017/20

lifecycle, for example, business requirements, system designs, and effective processes in the software development lifecycle.
code, test plans, test cases and documentation [10], [12]. A For example, Frank Elberzhager, Jürgen Münch and Danilo
systematic evaluation of these documents and artefacts occurs Assmann in [2] conducted a study to identify relationships
during this process. One of main software review process between software reviews and software testing. In the course of
benefits is that it can be used at the earliest software their work, three approaches how to define and direct the testing
development project phases [1], [5], [6]. There are many types process using defect detection at the project earliest phases were
of software review processes. The most popular types are analysed. As a result of this study, it was proved that there were
formal and informal reviews, inspection, walkthrough, relationships between software testing and software reviews.
technical reviews, peer reviews and management reviews. Information obtained in software review can be used to define
Despite the fact that all types of the software review have the and direct the software testing process.
same objective, there are differences between their In turn, Anurag Goswami, Gursimran S. Walia and Urvashi
implementation processes [6]. Rathod in [3] described how it was possible to improve an
Software formal reviews follow a formal process, which is ability of individual participants involved in the software
implemented in accordance with a structured and regulated review process to detect defects because the success of the
procedure. The most common type of formal reviews is software reviews strictly depended on qualification and skills
software inspection. Software inspection is the most of participants involved. As a result of this study, it was proved
comprehensive software review process that is implemented in that was necessary to keep in mind qualifications and skills of
strict compliance with the procedure, and it is led by the trained participants in the software reviews to obtain maximum results.
moderators. Software formal reviews, especially software Aybuke Autum, Hakan Petersson and Claes Wohlim in [1]
inspection, consist of six main steps: planning, kick-off, summarised theoretical information about software review
preparation, review meeting, rework and follow-up [1], [11]. process and software inspections within 25 years of its
During these six steps, the following activities take place: introduction. They described development areas of software
review session planning and preparation for it, analysis of the reviews and inspections, new methods that were invented
software artefacts and documents, review meeting, result during this time, and advantages and disadvantages of the
analysis and evaluation. More detailed information about software inspections. Chris Kemerer and Mark Paukl in [5]
software formal reviews and software inspection can be found investigated how the software review in the coding phase could
in the article “State-of-the-Art: Software Inspections after 25 impact and improve quality of the software.
Years” [1] and in the ISTQB [6] book.
Software informal reviews, in turn, are not documented and III. CASE STUDY DESIGN AND COLLECTED DATA
can be applied many times during the software development Many studies describe the benefits of the software reviews
process without any structured preparation and which occur during the requirements analysis and development
organisation [6]. The most common type of informal reviews is [3]–[5]. The present paper describes benefits of the test case
a walkthrough. Walkthroughs are led by the software artefacts reviews which occur in software testing. The following research
or document authors. An author guides the participants through questions are raised:
the documents according to his/her thought process to achieve • What role is played by communication and test case
a common understanding and gather feedback in order to reviews in software testing?
improve software product and artefacts [6]. • Can timely information availability and test case reviews
Technical reviews vary from quite informal to very formal improve software quality?
reviews and they are led by a trained moderator or a technical • Can test case reviews reduce the total testing time?
expert. It is often performed as a peer review without The case study has been conducted at the company which
management participation. Technical reviews have the same deals with the development, support, testing and maintenance
objectives as walkthroughs. In both cases, it is the fastest and of telecommunication and information technology services in
less expensive way how to get feedback about software, nine European countries: Latvia, Estonia, Lithuania, Sweden,
software artefacts and its quality in general [6]. the Netherlands, Germany, Austria and Croatia. The study has
The software reviews are based on communication between been conducted within one department which is responsible for
project development team members and all project stakeholders support and development of billing system “XXX”. New billing
[1], [6]. Communication and software reviews are the most system releases have been discussed and examined in the study.
important processes which should be taken into account not A new release of the billing system for one country rolls out
only at the software testing phase, but also during the all every 4–8 months. Since there are nine countries in total, there
software development life cycle. Communication is a “two- is roll-out of one or two new system releases each month. Here
way” process of information exchange, in which information is it is useful to note that business requirement analysis, software
received and understood by both participants [6]. Thoughtful testing, installation and software maintenance are performed by
communication is a gold key to successful project department employees, but software design, development and
implementation. Poor communication сan lead to confusion and coding are implemented by an outsourcing company. The case
misunderstanding. study analyses only software testing, including test planning,
In the software engineering literature, there are articles which analysis of business requirements and solution description, test
describe software review process as one of the most important case design and testing of new features as well as system

49

Unauthentifiziert | Heruntergeladen 29.01.20 13:42 UTC

 Information Technology and Management Science
_______________________________________________________________________________________________ 2017/20

preparation for a new release. Figure 1 displays the structure of 62 % – major defects and 13 % – minor defects. The average
software testing phase. For each release, it is planned to carry number of defects found in the release is 51 defects: 13 critical
out a 4-week testing phase. defects, 32 major defects and 6 minor defects.

35
31
Analysis of
Testing
business
Test case
30
requirements Test execution 25
planning desing
and solution 23
description
25

Quanity
20
15 15
15
Fig. 1. Software testing phase. 10
9
5 6 6
4 3
5
The case study has been organised into 3 steps: 1
0
1. Collecting statistics of 4 release testing before XXX 5.5 XXX 5.6.1 XXX 5.6.2 XXX 5.7
introducing the test case review; Release
2. Implementing the process of test case review;
3. Collecting statistics of 4 release testing when test case Critical Major Minor

reviews have been used.

The statistics were collected based on communication Fig. 2. Defect distribution by complexity, before software review process
implementation.
between testers and developers, business analysts, support
specialists, and weekly reports that have been delivered to Step 2. Implementation of the Test Case Review Process
stakeholders within a year. Detailed summary of case study
steps and collected data are provided further in this section. The test cases review process has been implemented in
software testing for the case study project. During the test cases
Step 1. Collection of Statistics (before) reviews have been checked if test cases are developed in
Table I and Fig. 2 summarise the quantity and complexity of accordance with defined requirements and standards. Several
defects found during testing of four releases before the test case checkpoints have been defined to support the review process.
reviews have been implemented. It is important to note that Example of checkpoints is given in Table II. The test case
Table I contains information about the defects which have been reviews are performed by another tester who did not write them,
found in the first new system release testing. It means that if but who has the same qualification as an author, or even better,
another country is rolling out a similar system release version, e.g., a test lead or a business analyst.
then only regressive testing is carried out. Table IV summarises defects, errors, gaps and problems and
All defects are divided into three levels of complexity: their occurrence reasons that have been found in the test cases
critical defects, major defects and minor defects. A critical level reviews. Many defects were timely discovered and eliminated
is assigned to a defect, which completely hampers or blocks the without any extra costs using information discovered in the test
system functionality. A major level is assigned to a defect, case reviews. The information also enables testers to enhance
which occurs when the functionality is functioning grossly their knowledge and better develop future test cases, which also
away from the expectations or not doing what it should be affect quality and speed of testing in the future software
doing, but a minor level is assigned to a graphic or grammatical releases. A total of 17 test case review sessions have been
defect. organised during an evaluation period.
The data analysis of defects shows that a total of 206 defects
have been found, where 25 % of them are critical defects,
TABLE I
NUMBER OF DEFECTS BEFORE SOFTWARE REVIEW PROCESS IMPLEMENTATION
Release Number of defects and its degree of complexity Total
1 week 2 week 3 week 4 week
XXX 4.8 12 (critical) 8 (critical) 3 (major) 2 (minor) 68
20 (major) 15 (major) 5 (minor)
3 (minor)
XXX 5.2 10 (critical) 18 (major) 1 (critical) 3 (major) 62
18 (major) 4 (major) 2 (minor)
1 (minor) 4 (minor)
XXX 5.3 3 (critical) 7 (major) 2 (major) 0 30
12 (major) 2 (minor) 4 (minor)
XXX 5.4 11 (critical) 6 (critical) 5 (major) 1 (critical) 46
10 (major) 10 (major) 2 (minor) 1 (major)

50

Unauthentifiziert | Heruntergeladen 29.01.20 13:42 UTC

 Information Technology and Management Science
_______________________________________________________________________________________________ 2017/20

TABLE II
CHECKLIST FOR TEST CASE REVIEW
No. Checklist Assessment (Yes/No) Remarks
1. Is the correct test case template being used?
2. Is the following information correct?
• References to business requirements;
• information about the author;
• creation time;
• an idea on the test cases;
• preconditions for test cases execution.
3. Was a product risk factor taken into account when test case execution conditions were defined?
4. Are the test cases able to cover all defined requirements?
5. Are external areas, which could affect the implementation of the requirement, identified and
included in the test cases?
6. Are equivalence classes identified? Are all possible equivalence classes included in the test
cases?
7. Are test data identified and included in the test cases?
8. Are boundary values, negative and invalid values identified and included in the test cases?
9. Are negative scenarios included in the test cases?
10. Are test steps defined in a correct and logical sequence?
11. Are the expected results defined for all test steps?
12. Is the expected result correctly identified?
13. Are test cases free of grammatical errors?
14. Are test cases developed consistently with use cases?

Step 3. Collection of Statistics (After)

IV. RESULTS
Table III and Fig. 3 summarise the number of defects and
Statistical data analyses before and after implementation of
complexity of release testing after the test cases review process
the test case review process show that the total number of
has been implemented.
defects has been reduced by 30 %. The approximate percentage
50 of critical, major and minor defects did not change but
45 43 decreased a possibility of finding defects at the end of the
40 38 testing when test execution was almost completed. Most of
35 defects are found during the first 2 weeks. Further, in this
30 26 section the results of the formulated research questions are
Quanity

25
20 21 discussed.
20 18

15 11
A. What Role Is Played by Communication and Test Case Reviews
10
10 7 6
in Software Testing?
5 3 2 The reviews are based on communication among not only
0
XXX 4.8 XXX 5.2 XXX 5.3 XXX 5.4
project development team members, but also among all project
Release stakeholders. In testing, the test case reviews can help detect
errors in the test cases and also business requirements.
Critical Major Minor
Timely elimination of defects reduces total project costs
Fig. 3. Defect distribution by complexity, after software review process because finding and fixing defects at the earliest project phases
implementation. are much cheaper than fixing such defects at the latest project
phases [2]. During the test case review, many problems with
The data analysis of defects shows that 145 defects have been business requirements were found. As the errors in the business
found, 23 % of them are critical defects, 65 % – major defects requirements are defect occurrence reasons in later software
and 11 % – minor defects. The average number of defects found products, it is necessary to think of a possibility of organising
in the release is 36 defects: 8 critical defects, 24 major defects business requirements reviews. If the developer has the
and 4 minor defects. opportunity to check the test cases while implementing a code,

51

Unauthentifiziert | Heruntergeladen 29.01.20 13:42 UTC

 Information Technology and Management Science
_______________________________________________________________________________________________ 2017/20

it is possible that this will help implement codes that may cause The availability of information can reduce confusion among
potential defects. During the test case reviews, problems with project team members and reduce the possibility of taking
communication between testers and developers were found. As incorrect decisions. Software reviews are based on
developers were external service providers and communication communication and availability of information. Timely
was held in English and mostly by emails, sometimes it took detection of errors and failures and their elimination during
more time to explain to developers where defects and problems software reviews increase software quality by reducing the
were. Table IV shows defects and mistakes found during the possibility of finding defects after software release.
test case reviews and their occurrence. Many defects are fixed During test case reviews, problems with information
without any additional costs. availability were found. Often important information is not
timely passed to the tester, so many errors and defects in the test
B. Can Timely Information Availability and Test Case Reviews
cases occur in that regard. Testers sometimes use incorrect and
Improve Software Quality?
outdated requirements and solution description versions, which
Information is the main exchange material in the project and in turn can result in some mistakes.
team interaction. The ongoing exchange of information allows
quickly finding necessary solutions and taking important C. Can Software Reviews Reduce the Total Testing Time?
decisions. Lack of information or incorrect information can Partially, software reviews can reduce the testing time which
cause misunderstanding among project team members. is required to verify if defects are fixed because many errors and
Misunderstanding of information increases the possibility of failures are already eliminated during the test case reviews. The
taking an incorrect decision or leads to errors in many project total testing time is influenced not only by the test cases
documents. Therefore, the lack of communication and reviews, but also by the human factor, force majeure,
information is one of the defect occurrence reasons in the emergency situations, the delivery time of new
software development. functionality etc.

TABLE III
NUMBER OF DEFECTS, AFTER SOFTWARE REVIEW PROCESS IMPLEMENTATION
Release Number of defects and its degree of complexity Total
1 week 2 week 3 week 4 week
XXX 5.5 4 (critical) 9 (major) 1 (minor) 0 22
6 (major) 2 (minor)
XXX 5.6.1 9 (critical) 10 (major) 4 (major) 3 (minor) 39
11 (major) 2 (minor)
XXX 5.6.2 5 (critical) 1 (critical) 5 (minor) 1 (minor) 35
12 (major) 11 (major)
XXX 5.7 10 (critical) 5 (critical) 1 (minor) 0 47
16 (major) 13 (major) 2 (major)

TABLE IV
MOST OFTEN DETECTED DEFECTS AND THEIR OCCURRENCE REASONS
Defect Occurrence reason
Incomplete test cases Insufficient knowledge of the developed functionality or software
Lack of negative test cases Insufficient knowledge of the software testing (methods, techniques, principles etc.)
Lack of data in the requirements
Changes to the requirements after test case development was finished
Lack of test data Lack of data in the requirements
Inappropriate and incorrect test data Insufficient knowledge of the developed functionality or software
Incorrect expected results Insufficient knowledge of the developed functionality or software
Grammatical mistakes No check of the existence of grammatical errors was made
Incomplete test execution results Data about the expected results and defects are not updated after each test execution
Information about defects is not updated Data about the expected results and defects are not updated after each test execution
Test cases are not updated if changes are made to requirements Lack of communication. Information about changes in requirements is not passed to
testers

52

Unauthentifiziert | Heruntergeladen 29.01.20 13:42 UTC

 Information Technology and Management Science
_______________________________________________________________________________________________ 2017/20

V. CONCLUSION [4] K. Holl and F. Elberzhager, “Mobile Application Quality Assurance:

Reading Scenarios as Inspection and Testing Support,” 2016 42th
In this case study, the test case review process has been Euromicro Conference on Software Engineering and Advanced
organised and implemented in software testing. Implementation Applications (SEAA), Aug. 2016. https://doi.org/10.1109/seaa.2016.11
[5] C. F. Kemerer and M. C. Paulk, “The Impact of Design and Code Reviews
of software reviews is one of the project success factors. During on Software Quality: An Empirical Study Based on PSP Data,” IEEE
the test cases reviews, many errors are detected and corrected Transactions on Software Engineering, vol. 35, no. 4, pp. 534–550, Jul.
not only in the test cases, but also in the requirements and 2009. https://doi.org/10.1109/tse.2009.27
[6] A. Spillner, T. Linz and H. Schaefer, Software Testing Foundations:
software code, which in turn affect software quality and project A Study Guide for the Certified Tester Exam, Rocky Nook, 2007.
costs. As a result, the number of critical defects and a total [7] “What are the principles of testing?” [Online]. Available:
number of defects for separate system releases are reduced [2], http://istqbexamcertification.com/what-are-the-principles-of-testing/
[5], [12]. [8] Q. Shan, G. Rong, H. Zhang, G. Liu, and D. Shao, “An Empirical
Evaluation of Capture-Recapture Estimators in Software Inspection,”
The review process of project documents (requirements, 2015 24th Australasian Software Engineering Conference, Sep. 2015.
system designs, software code, test plans, management plans https://doi.org/10.1109/aswec.2015.17
etc.) helps improve and achieve maximum results. Defects [9] H. Potter, M. Schots, L. Duboc, and V. Werneck, “InspectorX: A game
for software inspection training and learning,” 2014 IEEE 27th
found during test case reviews helped localise problematic Conference on Software Engineering Education and Training (CSEE&T),
areas in the software development lifecycle [3]–[5]. Apr. 2014. https://doi.org/10.1109/cseet.2014.6816782
To obtain a maximal result from reviews, it is necessary to [10] F. Salger, G. Engels, and A. Hofmann, “Inspection effectiveness for
different quality attributes of software requirement specifications: An
take into account participants’ skills and knowledge, especially industrial case study,” 2009 ICSE Workshop on Software Quality, May
qualifications, skills and knowledge when selecting reviewers. 2009. https://doi.org/10.1109/wosq.2009.5071552
The reviewers who are inexperienced, i.e., do not know [11] N. Hashemitaba and S. H. Ow, “Generative Inspection: An Intelligent
business logic and system functionality, cannot find errors and Model to Detect and Remove Software Defects,” 2012 Third
International Conference on Intelligent Systems Modelling and
can even make new mistakes. Similar conclusions are made in Simulation, Feb. 2012. https://doi.org/10.1109/isms.2012.48
[3], [9], [11]. [12] D. L. Parnas and M. Lawford, “The role of inspection in software quality
In general, the results and the accuracy of analysis obtained assurance,” IEEE Transactions on Software Engineering, vol. 29, no. 8,
pp. 674–676, Aug. 2003. https://doi.org/10.1109/tse.2003.1223642
in the paper are limited to the defined case study. To interpret
the obtained knowledge as a general approach that can be used Oksana Petunova has earned her BSc. degree in Information Technology from
in software testing, additional case studies on other projects are Riga Technical University, Latvia (2015). The current position is Information
necessary. Regarding this case study, in future the reviews are System Test Specialist at Tele2 SSC. She holds ISTQB Certified Tester
certificate.
planned to be implemented in the business requirement analysis E-mail: [email protected]
and development processes.
Solvita Bērziša holds a Doctoral Degree and is an Assistant Professor at the
Institute of Information Technology of Riga Technical University (Latvia). She
REFERENCES obtained her Dr. sc. ing. (2012), BSc. (2005) and MSc. (2007) degrees in
[1] A. Aurum, H. Petersson, and C. Wohlin, “State-of-the-art: software Computer Science and Information Technology from Riga Technical
inspections after 25 years,” Software Testing, Verification and Reliability, University. Her main research fields are IT project management,
vol. 12, no. 3, pp. 133–154, 2002. https://doi.org/10.1002/stvr.243 implementation and application of project management information systems, as
[2] F. Elberzhager, J. Münch, and D. Assmann, “Analyzing the relationships well as project data analytics, big data and data science. She also works as a
between inspections and testing to provide a software testing focus,” Team Lead at Accenture Latvia. She holds a PMP and CBAP certificates and
Information and Software Technology, vol. 56, no. 7, pp. 793–806, Jul. was awarded the IPMA Outstanding Research Contribution of a Young
2014. https://doi.org/10.1016/j.infsof.2014.02.007 Researcher 2013. She is a Member of PMI and Latvian National Project
[3] A. Goswami, G. S. Walia, and U. Rathod, “Using Learning Styles to Staff Management Association.
and Improve Software Inspection Team Performance,” 2016 IEEE E-mail: [email protected]
International Symposium on Software Reliability Engineering Workshops
(ISSREW), Oct. 2016. https://doi.org/10.1109/issrew.2016.38

53

Unauthentifiziert | Heruntergeladen 29.01.20 13:42 UTC