Executing Test

(Software Reliability)
(SE-308)

By Priya Singh (Assistant Professor, Dept of SE, DTU)

1. Background
• Types of software reliability engineering test-
1. reliability growth test and
2. certification test.
• These types are not related to phases of test such as unit test, subsystem test,
system test, or beta test, but rather to the objectives of test.

By Priya Singh (Assistant Professor, Dept of SE, DTU)

2. Reliability Growth Test
• The main objective of reliability growth test is to find and remove faults.
• During it, you use software reliability engineering to estimate and track failure intensity.
Testers and development managers apply the failure intensity information to guide
development and to guide release.
• You typically use a reliability growth test for the system test phase of software you
develop in your own organization.
• You can also use it in beta test if you are resolving failures (removing the faults causing
them) as you test.
• To obtain "good" (with moderate ranges of uncertainty) estimates of failure intensity, you
need a minimum number of failures in your sample, often 10 to 20.
• You may follow reliability growth test by certification test as a rehearsal if your customer
will be conducting an acceptance test.

By Priya Singh (Assistant Professor, Dept of SE, DTU)

3. Certification Test
• With the certification test, you make a binary decision: accept the software, or reject the
software and return it to its supplier for the rework.
• In the certification test, you require a much smaller sample of failures. In fact, you can
make decisions without any failures occurring if the period of execution without failure is
sufficiently long.
• We generally use certification tests only for load tests (not feature or regression tests).
• If we are simply integrating the product from components, we will conduct only a
certification test of the integrated base product and its variations and supersystems.
• Certification test does not involve debugging.
• There is no attempt to "resolve" failures you identify by determining the faults that are
causing them and removing the faults.
• The system must be stable.
• No changes can be occurring, either due to new features or fault removal.

By Priya Singh (Assistant Professor, Dept of SE, DTU)

• Feature test is the test that executes all the new test cases for the new operations of the
current release, independently of each other, with interactions and effects of the field
environment minimized.
• Feature test can start after you implement the operation you will test, even if the entire
system is not complete.
• Load test is the test that executes all valid test cases from all releases together, with full
interactions.
• Invocations occur at the same rates and with the same other environmental conditions as
will occur in the field.
• Its purpose is to identify failures resulting from interactions among the test cases,
overloading of and queueing for resources, and data degradation.
• Among these failures are concurrency failures (deadlock, race, etc.) where different test
cases are changing the same variable.
• Acceptance tests and performance tests are types of load test

By Priya Singh (Assistant Professor, Dept of SE, DTU)

• Regression test is the test that executes a subset of all valid test cases of all releases at
each system build with significant change, independently of each other, with interactions
and effects of the field environment minimized.
• Whether "significant change" occurred requires a judgment call; one criterion might be
the percent of code changed.
• The subset can consist of all the test cases. When less that the total set, the sample
changes for each build and it includes all the critical test cases.
• Its purpose is to reveal functional failures caused by faults introduced by program
changes since the prior build. Note that this type of test does not check the effects of
indirect input variables.

By Priya Singh (Assistant Professor, Dept of SE, DTU)

4. Steps in executing tests
• In executing the test, we will use the test cases and test procedures developed in preparing
for test activity in such a way that we realize the goal of the efficient test that we desire.

• Executing test involves three main sub-activities:

i. determining and allocating test time,
ii. invoking the tests, and
iii. identifying failures that occur.

• You will use the system failure data in the Guiding Test activity

By Priya Singh (Assistant Professor, Dept of SE, DTU)

4.1 Planning and allocating test time for the current release
What is test time?
• Test time is the time required on test units to set up, execute, record results of, and clean
up after tests whose purpose is to find previously undiscovered failures or the absence
thereof.
• It includes immediate preparation such as setup and immediate follow-up such as cleanup
and identification of failures but it doesn't include long-term preparation such as the
development of test cases and procedures.
• It does not include the time required for failure resolution (finding and removing the
faults, causing the failures, and demonstrating that the failures have been resolved).

By Priya Singh (Assistant Professor, Dept of SE, DTU)

Planning test time:
• To plan the total amount of test time you will use, start by determining the test time you
need.
• We can determine the failure intensity reduction we must achieve through system test,
which depends on the failure intensity objective and the software reliability strategies we
implement.
• We do not presently know how to predict the amount of test needed to achieve this failure
intensity reduction.
• It certainly depends heavily on how closely the operational profile we use in the system
test matches the operational profile we will find in the field and how successful we are
inaccurately identifying equivalence classes.
• At present, we choose the amounts of time we think we need as a result of past experience
with previous releases or similar products.
• The best indicator of similarity of the product appears to be a similar failure intensity
reduction that is to be achieved through a system test.

By Priya Singh (Assistant Professor, Dept of SE, DTU)

Estimating test time:
• Next estimate your test time capacity by multiplying the system test period by the number
of test units, where a test unit is a facility needed and available for test.
• One test unit includes all the processors and hardware needed and is available to test an
associated system.
• Sometimes we reconfigure groups of processors during test, resulting in a different
number of test units at different times.
• In this case, use an average value for the test period.
• If you have multiple test units, test can proceed at a more rapid rate

By Priya Singh (Assistant Professor, Dept of SE, DTU)

Finalize the test time
• We then decide on the test time we will plan for.
• If the test time we need and test time capacity are reasonably close to one another, we will
plan based on the test time capacity.
• If they differ greatly, we must add test units or increase the system test period.
• You only negotiate for more test units or a longer system test period when the need is
substantially greater than capacity because estimation algorithms are approximate and
may be challenged.
• You typically estimate the amount of test time you will plan late in the requirements
phase, when the general outline of the requirements becomes clear.

By Priya Singh (Assistant Professor, Dept of SE, DTU)

Allocation of test time for a release proceeds in two steps:
1. Allocation among the associated systems to be tested (base product, variations, and
supersystems of both base product and variations)
2. Allocation among feature, regression, and load test for reliability growth test of each
base product or variation, after assigning time for certification test of each of these
associated systems that need it (for example, if a customer is performing acceptance
tests)

Note that the test of supersystems is entirely the certification test.

The certification test of any associated system consists entirely of load test.

By Priya Singh (Assistant Professor, Dept of SE, DTU)

4.2 Invoking test
• Software reliability engineering test starts after the units of the system being tested have
been tested or otherwise verified and integrated so that the operations of the system can
execute completely.
• If this condition isn't at least approximately met, it will be difficult to drive tests with the
operational profile and create a reasonable representation of field conditions.
• We can test the base product and its variations in parallel. Each is followed by the test of
its supersystems.
• For the base product and each of its variations, test starts with feature test followed by
load test, with regression test at each build during the load test period.
• Reliability growth of the developed software is tracked during the test of the base product
and each variation. There may also be a period of certification test at the end of each of
the customers do acceptance test.

By Priya Singh (Assistant Professor, Dept of SE, DTU)

Feature Tests
• During feature test, you invoke each test case after the previously invoked test case has
completed execution.
• Each test case is selected randomly from the set of new test cases for the release.
• Do not replace the test case in the group available for selection after execution. You must
provide setup and cleanup for each execution.
• Feature tests will slightly overemphasize critical operations, but any resulting distortion of
the failure intensity estimates will usually be negligible.
• Since we have selected test cases in accordance with the operational profile, feature test
will be a true representation of field operation except that failures due to data degradation,
traffic level, environmental conditions, and interaction among operations will not occur.

By Priya Singh (Assistant Professor, Dept of SE, DTU)

Load Test
• During load test, you invoke test cases with the test procedure, pacing them at an average
rate based on the traffic level specified for the current time.
• This approach flushes out failures related to traffic level as a proportion of capacity.
Invocation of test cases at different times results in test cases being executed under a wide
variety of states of data degradation.
• Usually you invoke test cases at random times but you can invoke them at fixed times.
• We provide for test case invocation to occur at random times to ensure that the occurrence
probabilities are stationary (unchanging with time).
• We do not want to duplicate runs, because a duplicate run provides no new information
about failure behavior of the system. Thus duplication is wasteful of testing resources.

By Priya Singh (Assistant Professor, Dept of SE, DTU)

• Invoking test cases at random times will cause them to execute with different indirect
input variables, resulting in the runs being different.
• The number of test cases invoked will be determined automatically, based on the test time
allocated and the occurrence rate.

By Priya Singh (Assistant Professor, Dept of SE, DTU)

Regression Tests
• In regression test, you invoke each test case after the previously-invoked test case has
completed execution.
• You choose a subset of test cases for each build, consisting of all valid critical test cases
and a specified number of test cases chosen randomly from all valid noncritical test cases,
with the latter test cases not replaced until exhausted.
• The subset must be sufficiently large such that all valid test cases will be executed over
the set of builds for the release.
• A valid test case is one that tests an operation that is valid or is still active and considered
a feature of the current release.

By Priya Singh (Assistant Professor, Dept of SE, DTU)

• You can select a subset consisting of the entire set of test cases if you wish; however, the
testing that results will be inefficient and can be justified only for low failure intensity
objectives.
• Since we have selected test cases in accordance with the operational profile, regression
test will be a true representation of field operation except that failures due to data
degradation, environmental conditions, and interaction among operations will not occur.
• Operational Profile- Testing driven by an operational profile is very efficient because it
identifies failures (and hence the faults causing them), on average, in the order of how
often they occur. This approach rapidly reduces failure intensity as test proceeds, and the
faults that cause frequent failures are found and removed first. Users will also detect
failures on average in order of their frequency, if they have not already been found in test.

By Priya Singh (Assistant Professor, Dept of SE, DTU)

4.3 Identifying failures
• We identify system failures by analyzing test output promptly for deviations, detennining
which deviations are failures, and establishing when the failures occurred.
i. Analyzing test output for deviations
ii. Determining which deviations are failures
iii. Establishing when failures occurred

By Priya Singh (Assistant Professor, Dept of SE, DTU)

4.3.1 Analyzing test output for deviations
• A deviation is any departure of system behavior in execution from expected behavior. It
differs from a failure in that a failure must make a user unhappy.
• There are two reasons why we developed the concept of "deviation" in addition to that of
"failure": deviations can often be detected automatically but failures usually require the
judgment of a person, and deviations provide a nice way of describing fault tolerance.
• We can usually automatically detect several types of deviations. First, many standard
types are readily detected by the operating system: inter-process communication failures,
illegal memory references, return codes indicating deviant behavior, deadlocks, resource
threshold overruns, and process crashes or hangs.
• Then we have easily recognized application behaviors (for example, incomplete calls or
undelivered messages in telecommunications systems).
• Finally, you can examine the output automatically with built-in audit code or an external
checker.

By Priya Singh (Assistant Professor, Dept of SE, DTU)

4.3.2 Determining which deviations are failures
• In order to determine which deviations are failures, note that a deviation must violate user
needs to be a failure. Manual analysis of deviations is often necessary to determine
whether they violate user requirements and hence represent failures.
• For example, identifying an incorrect display is hard to automate. However, usually
failures of higher severity involve easily observable effects that would unquestionably
violate user requirements.
• The deviations in fault-tolerant systems are not failures if the system takes action to
prevent them from violating user needs. But intolerance of deviations by a fault-tolerant
system that is supposed to be tolerant of them may be a failure.

By Priya Singh (Assistant Professor, Dept of SE, DTU)

4.3.3 Establishing when failures occurred
• All the failures for an associated system are grouped together in a common sequence in
the order in which they occur, regardless of whether they occurred in a feature test,
regression test, or load test.
• We will use as our measuring unit of when a failure occurred the common reference unit
(natural or time units) chosen for the project to set failure intensity objectives and
describe failure intensities.
• There is one exception. If you are using an operating time unit, and the average computer
utilization varies from one period compared to a failure interval to another period
comparable to a failure interval, the time intervals measured will be influenced by this
variation.

By Priya Singh (Assistant Professor, Dept of SE, DTU)