BITP 3253:

VERIFICATION &
VALIDATION
Week 3: Testing Process

Fakulti Teknologi Maklumat dan Komunikasi

 Overview
• Planning and Control
• Analysis and Design
• Implementation and Execution
• Evaluating test exit criteria and Reporting
• Test closure activities

Fakulti Teknologi Maklumat dan Komunikasi

 Test Process: Critical Testing
Process
Understand the testing effort
Discover the context of testing
Analyze the quality risks Plan
Estimate the testing
Assemble the people and test
Plan the testing
Build the test team
Design and implement a test
Guide adoption and improvement
system
Report any bugs Perfect Prepare
Report test results
Manage changes

Perform Do the testing and gather the result

Obtain a test release
Run and tract the tests

The test engineer’s role focuses on some of the activities, not all,
depending on how roles are defined. However, the effective and efficient
test engineer must understand how the test process works and how it fits
into the overall project from a bi picture perspective

Fakulti Teknologi Maklumat dan Komunikasi

 Fundamental Test Process

The fundamental test process:-

• Planning and Control
• Analysis and design
• Implementation and execution
• Evaluating test exit criteria and reporting
• Test closures activities
• These steps may overlap, be concurrent, an/or
iterate

Fakulti Teknologi Maklumat dan Komunikasi

 Planning and Control

Planning: Is the activity of verifying the mission of

testing, defining the objectives of testing and the
specification of test activities in order to meet
the objectives and mission.

Control: Is the ongoing activity of comparing actual

progress against the plan, and reporting the
status, including deviation from plans

Fakulti Teknologi Maklumat dan Komunikasi

 Planning and Control Activities
Planning
• Determine test scope, test levels,
risks, objectives, strategies, test Control
exit criteria • Measure and analyze results
• Determine required test resources • Monitor and document
(who, what, how and when) progress, coverage and test
• Schedule test activities exit criteria
• Set the level of detail for test • Initiate corrective actions
cases, procedures in order to • Make decision
provide enough information to
support reproducible test
preparation and execution Many planning and control
activities involve obtaining
• Select test monitoring, controlling,
project team and project
and reporting metrics, charts and
management agreement,
reports (deliverables)
support and consensus.

Fakulti Teknologi Maklumat dan Komunikasi

 Developing Test Plans
• Why write (and update) test plans?
• Confront challenges, crystallize thinking, adapt to change
• Communicate plan to testers, peers and managers
• Consider multiple test plans when tests have..
• Different time periods (e.g. phases and levels)
• Different methodologies and tools (e.g. performance and
functionality)
• Different objectives (e.g. system test and beta test)
• Different audiences (e.g. hardware and software test )
• ..But you may want Master Test Plan
• Circulate one or two drafts
• Promotes early feedback and discussion
• Prevents wasted time if you’re on the wrong track
Fakulti Teknologi Maklumat dan Komunikasi
 IEEE 829 Test Plan
• A test plan is subproject plan for the testing part
of a project, and includes the sections shown in
the next slide
• You can adapt the IEEE 829 outline for use for
each detail (e.g. level or phase) test plan as well
as the master test plan
• You can create you own template or outline, too
• Test planning influences (and is influenced by)
test policy of the risk organization, the scope of
testing, objectives, risks, constraints, criticality,
testability, and the availability of resources
Fakulti Teknologi Maklumat dan Komunikasi
 IEEE 829 Test Plan Outline

• Test plan identifier • Test deliverables (e.g. reports

• Introduction , charts, etc)
• Test items (i.e. what’s • Test tasks (or at least key
delivered for testing) milestones)
• Features to be tested • Environmental needs
• Features not to be tested • Responsibilities
• Approach (strategies, • Staffing and training needs
organization, extent of testing) • Schedule
• Item pas/fail criteria • Risks and contingencies
• Test criteria (e.g. entry, exit, (quality [product] and project
suspension and resumption) risks)
• Approvals

Fakulti Teknologi Maklumat dan Komunikasi

 Entry Criteria
• Entry criteria measure whether the system is
ready for a particular test phase.
• Deliverables ready?
• Lab ready?
• Teams ready?
• These tend to become increasingly rigorous as
the phase proceed

Fakulti Teknologi Maklumat dan Komunikasi

 Sample Entry Criteria

System Test can begin when:- 6. Less thank 50 must-fix bugs (per Sales,
1. Bug tracking and test tracking system are Marketing and Customer Service) are
in place open against the release
2. All components are under formal, 7. The Development Teams provide
automated configuration and release software to the Test Team 3 business
management control. days prior to starting System Test
3. The Operations team has configured the 8. The Test Team completes a 3 day
System Test Server environment “smoke test” and reports on the results
including all target hardware components 9. The Project Management Team agrees
and subsystems. The Test team has in a System Test Phase Entry Meeting
been provided with appropriate access to to proceed. The following topics will be
these systems resolved in the meeting:-
 Whether code is complete
4. The Development Teams have completed  Whether unit-testing is complete
all features and bug fixes scheduled for  Assign a target fix date for any known “must fix”
release bugs (no later than 1 week after System Test
Phase Entry)
5. The Development Teams have unit-tested
all features and bug fixes schedule for
release

Fakulti Teknologi Maklumat dan Komunikasi

 Continuation Criteria

• Continuation criteria measure whether testing can

efficiently, effectively proceed
• Test environment problems
• Test-blocking bugs in system under test
• “Continuation criteria” is a polite way of saying
“stopping criteria” in the reverse
• Stopping a test phase is seldom popular

Fakulti Teknologi Maklumat dan Komunikasi

 Sample Continuation Criteria

System Test will continue if:- 3. The open bug backlog (“quality
1. All software released to the Test gap”) remains less than 50. The
Team is accompanied by Release daily and rolling closure periods
Notes remain less than 14 days (on
average, bugs are fixed within two
2. No change is made to the system,
weekly release cycles)
whether in source code,
configuration files, or other setup 4. Twice-weekly bug review meetings
instructions or processes, without an occur until System Test Phase Exit
accompanying bug report. Should a to manage the open bug backlog
change be made without a bug and bug closure times
report, the Test Manager will open
an urgent bug report requesting
information and escalate to his
manager

Fakulti Teknologi Maklumat dan Komunikasi

 Exit Criteria

• Continuation criteria measure whether the test

phase can be deemed complete
• Thoroughness measures, such as coverage of code,
functionality or risk
• Estimates of defect density or reliability measures
• Cost
• Residual risks, such as defects not fixed or lack of
test coverage in certain areas
• Schedules such as those based on time to marker
• Remember! These are business decision
Fakulti Teknologi Maklumat dan Komunikasi
 Sample Exit Criteria

System Test will continue if:- 6. The Test Team has checked that all
1. NO changes (design/code/features), issues in the bug tracking system are
except to address System Test defects, either closed or deferred, and where
occurred in the prior 3 weeks appropriate, verified by regression and
confirmation testing
2. NO panic, crash, halt, wedge, unexpected
process termination, or other stoppage of 7. The test metrics indicate product stability
processing has occurred on any server and reliability; completion of all planned
software or hardware for the previous 3 tests; adequate coverage of the critical
weeks quality risks
3. No client systems has become inoperable 8. The Project Management Team agrees
due to a failed update during System Test that the product, ad defined during the
final cycle of System Test, will satisfy the
4. The Test Team has executed all the
customer’s reasonable expectations of
planned tests against the GA-candidate
quality
software
9. The Project Management Team holds a
5. The Development Teams have resolved
System Test Phase Exit Meeting and
all “must-fix” bugs per Sales, Marketing
agrees that we have completed System
and Customer Service
Test

Fakulti Teknologi Maklumat dan Komunikasi

 Analysis and Design

Test analysis and design : Is the activity where

general testing objective are transformed into
tangible test conditions and test cases.

Fakulti Teknologi Maklumat dan Komunikasi

 Analysis and Design Activities

Analysis Design
• Review test basis (e.g. • Design and prioritize
requirements or design combinations of test data,
specifications, network/system actions, and expected results
architecture, quality risks) • Identify the test data needed
• Identify and prioritize test for test conditions and cases
conditions, test requirements, • Design the test environment
or test objectives and required
• Identify infrastructure tools
test data based on analysis of
test items (e.g. their behavior,
specification, and structure)
• Evaluate testability of the
requirements and system

Fakulti Teknologi Maklumat dan Komunikasi

Implementation and Execution

Implementation and execution: Is the activity

where test procedures of scripts are specified
by combining test cases in a particular order ad
including any other information needed for test
execution, the environment is set up and tests
are run

Fakulti Teknologi Maklumat dan Komunikasi

 Implementation and Execution
Activities
Implementation Execution
• Develop, implement, and • Execution test cases (manual
prioritize test cases, create or automated)
data, write procedures • Log test results, and the
• Create test harness, scripts versions of the software under
• Organize test suites and test, test tools and the testware
sequences of test procedures • Compare actual and expected
• Verify the test environment results
• Report and analyze incidents
• Repeat corrected and/or
updated tests
• Run confirmation and/or
regression tests

Fakulti Teknologi Maklumat dan Komunikasi

 Evaluating Exit Criteria and
Reporting
Evaluating exit criteria: Is the activities where test
execution is assessed against the defined
objectives.
This should be done for each test level.

Fakulti Teknologi Maklumat dan Komunikasi

 Evaluating Exit Criteria
Major Activities
• Check test logs against the exit criteria specified
in the test plan
• Assess if more tests are needed or if the exit
criteria specified should be changed
• Write a test summary report for stakeholders

Fakulti Teknologi Maklumat dan Komunikasi

 Test Reporting

• Summarize/analyze the test results

• Key events (e.g. meeting exit criteria)
• Analysis (for recommendations, guidance) of..
• Cost/benefit of more testing
• Outstanding risks
• Level of confidence

• Metrics are gathered to access:

• Test objectives adequacy for test level
• Test approaches adequacy
• Testing effectiveness per objectives
Fakulti Teknologi Maklumat dan Komunikasi
IEEE 829 Test Summary Report
• A test summary report describes the results of a given level
or phase of testing, and includes the following sections
• Test summary report identifier
• Summary (e.g. what was tested, what the conclusions
are, etc.)
• Variances (from plan, cases, procedures)
• Comprehensive assessment
• Summary of results (e.g. final metrics, counts)
• Evaluation (of each test item pass/fail criteria)
• Summary of activities (resource use, efficiency, etc.)
• Approvals
• Maybe delivered within or at end of a test level
Fakulti Teknologi Maklumat dan Komunikasi
Sample Summary Report

Fakulti Teknologi Maklumat dan Komunikasi

Sample Summary Report

Fakulti Teknologi Maklumat dan Komunikasi

 Closure

Test closure: Is the activities of collecting data from

completed test activities to consolidate
experience, testware, facts and numbers.

Fakulti Teknologi Maklumat dan Komunikasi

Thank You

Fakulti Teknologi Maklumat dan Komunikasi