BITP 3213:

Software Testing & Quality

Assurance
Week 6: Test Management

Fakulti Teknologi Maklumat dan

Komunikasi
 Overview

• Test Organization
• Test Planning and Estimation
• Test Progress Monitoring and Control
• Configuration Management
• Risk and Testing
• Bug or Incident Management

Fakulti Teknologi Maklumat dan

Komunikasi
Test Organization – What is the
“What is the mission with respect to the
Test
Team’s
organization, Job?and the product?”
the project

“The process consisting of all life cycle activities, both

static and dynamic, concerned with planning,
preparation and evaluation of software product
and related work products to determine that they
satisfy specified requirements, to demonstrate
that they are fit for purpose and to detect
defects.”

Fakulti Teknologi Maklumat dan

Komunikasi
Test Organization - What Is the Test
Team’s Job?
• Test / quality control
• Quality risk management service
• Assessor of quality service - Quality assessment
• Quality Assurance
• Test/QC Management
• Ensuring product quality through process
• Make sure you’re competent, staffed, and
politically supported for your role.

Fakulti Teknologi Maklumat dan

Komunikasi
 Test Organization - Levels of
Independence
• No independent testers, developers test their own code
• Independent testers within the development teams
• Independent test team or group within the organization,
reporting to project management
• Independent testers from the business organization or
user community
• Independent test specialists for specific test targets such as
usability testers, security testers or certification testers
• Independent testers outsourced or external to the
organization

Fakulti Teknologi Maklumat dan

Komunikasi
 The Value of Independence
Testing
• Given complex and/or critical application, one
needs..
• Multiple levels of testing
• Some or all of the levels done by independent
testers
• Best practices observed from best ISTQB
clients:-
• Low independence testing at lower test levels
• Higher independence testing at higher levels of
testing

Fakulti Teknologi Maklumat dan

Komunikasi
 Independence Considerations
• Benefits • Potential Traps
• See more, other and • Isolation from the
different defects development team
• Anything in doubt, they • Seen as bottleneck
will consider it as bug • Programmers lose sense
• Verify assumption of of responsibility for
specifications and quality
implementation
• Credible assessment of
quality
• Tester career path
Fakulti Teknologi Maklumat dan
Komunikasi
 Test Manager
• Devise test strategies, plans • Ensure configuration
• Write/ review test policy management of test ware
• Consult on testing for other • Ensure traceability
project activities • Measure test progress, evaluate
• Test estimation the quality of the testing and
• Test resource acquisition the product
• Lead specification, preparation, • Plan any test automation
implementation, and execution • Select tools and organize any
tests tester training
• Monitor and control the test • Ensure implementation of test
execution environment
• Adapt the test plan based on test • Schedule tests
result • Write test summary report

Fakulti Teknologi Maklumat dan

Komunikasi
• Review and contribute to test plans
• Analyze, review and assess user requirements,
specifications Tester
• Create test suites, cases, data and procedures
• Set up the test environment
• Implement tests on all test levels
• Execute and log the tests, evaluate results and document
problems found
• Monitor testing using the appropriate tools
• Automate tests
• Measure performance of components and systems
• Review each others’ tests
Fakulti Teknologi Maklumat dan
Komunikasi
 Refining the Tester Position
• Test engineers • Test technicians
• Technical peers of programmers • Skilled and experienced tester
• Chose testing as a specialty • Maybe an aspiring test engineer
• Write test cases, organize test • Runs tests
suites • Report bugs
• Create, customize and use • Update test status
advances test tools • Other team members
• Have unique skills • System and database admin
• Releases and configuration
The right positions for your team engineers
depends on the skills and positions • Test toolsmiths
across the test team-skills are
complimentary and the wholecan be
greater than the sum of the parts

Fakulti Teknologi Maklumat dan

Komunikasi
 Using Amateur Testers

• On many projects, amateurs testers (people who do not test

for their living) are used as part of the team
• Such people typically include project managers, quality
managers, programmers, business and domain experts, or
infrastructure or IT operators
• Such a test team often possesses a strong skills in some
areas (business domain/technology), but weak skills in
others, and no skills or substantial experience with testing
• Test is a special field, with special skills

Fakulti Teknologi Maklumat dan

Komunikasi
Test Organization - Balancing the Skills
• Good test teams have right mix of skills based
on tasks and activities
• Application (business) domain expert
• Understands intended behavior
• Skilled tester
• Knows quality risks and test techniques
• Technical guru
• Aware of technical issues and limitations

What is the right mix for your project??

Fakulti Teknologi Maklumat dan

Komunikasi
 Test Planning - Developing Test Plans
• Why write (and update) test plans?
• Confront challenges, crystallize thinking, adapt to change
• Communicate plan to testers, peers and managers
• Consider multiple test plans when tests have..
• Different time periods (e.g. phases and levels)
• Different methodologies and tools (e.g. performance and functionality)
• Different objectives (e.g. system test and beta test)
• Different audiences (e.g. hardware and software test )
• ..But you may want Master Test Plan
• Circulate one or two drafts
• Promotes early feedback and discussion
• Prevents wasted time if you’re on the wrong track

Fakulti Teknologi Maklumat dan

Komunikasi
 Test Planning Activities
• Define test approach, test levels
• Integrate, coordinate testing into the life cycle
• Decide who, what, when, how of testing
• Assign resources for test tasks
• Define the test documentation
• Set the level of detail for test cases, procedures in
order to provide enough information to support
reproducible test preparation and execution
• Select test monitoring, controlling, and reporting
metrics, charts and reports (deliverables)

Fakulti Teknologi Maklumat dan

Komunikasi
 IEEE 829 Test Plan
• A test plan is subproject plan for the testing part
of a project, and includes the sections shown in
the next slide
• You can adapt the IEEE 829 outline for use for
each detail (e.g. level or phase) test plan as well
as the master test plan
• You can create you own template or outline, too
• Test planning influences (and is influenced by)
test policy of the risk organization, the scope of
testing, objectives, risks, constraints, criticality,
testability, and the availability of resources

Fakulti Teknologi Maklumat dan

Komunikasi
 IEEE 829 Test Plan Outline

• Test plan identifier • Test deliverables (e.g. reports ,

• Introduction charts, etc)
• Test items (i.e. what’s delivered • Test tasks (or at least key
for testing) milestones)
• Features to be tested • Environmental needs
• Features not to be tested • Responsibilities
• Approach (strategies, • Staffing and training needs
organization, extent of testing) • Schedule
• Item pas/fail criteria • Risks and contingencies (quality
• Test criteria (e.g. entry, exit, [product] and project risks)
suspension and resumption) • Approvals

Fakulti Teknologi Maklumat dan

Komunikasi
 Entry Criteria
• Entry criteria measure whether the system is
ready for a particular test phase.
• Deliverables ready?
• Lab ready?
• Teams ready?
• These tend to become increasingly rigorous
as the phase proceed

Fakulti Teknologi Maklumat dan

Komunikasi
 Sample Entry Criteria
• System Test can begin when:- 6. Less thank 50 must-fix bugs (per Sales,
1. Bug tracking and test tracking system are in Marketing and Customer Service) are open
place against the release
2. All components are under formal, automated 7. The Development Teams provide software
configuration and release management to the Test Team 3 business days prior to
control. starting System Test
3. The Operations team has configured the 8. The Test Team completes a 3 day “smoke
System Test Server environment including all test” and reports on the results
target hardware components and 9. The Project Management Team agrees in a
subsystems. The Test team has been System Test Phase Entry Meeting to
provided with appropriate access to these proceed. The following topics will be
systems resolved in the meeting:-
4. The Development Teams have completed all  Whether code is complete
 Whether unit-testing is complete
features and bug fixes scheduled for release
 Assign a target fix date for any known “must fix” bugs
5. The Development Teams have unit-tested all (no later than 1 week after System Test Phase Entry)
features and bug fixes schedule for release

Fakulti Teknologi Maklumat dan

Komunikasi
 Continuation Criteria

• Continuation criteria measure whether

testing can efficiently, effectively proceed
• Test environment problems
• Test-blocking bugs in system under test
• “Continuation criteria” is a polite way of
saying “stopping criteria” in the reverse
• Stopping a test phase is seldom popular

Fakulti Teknologi Maklumat dan

Komunikasi
 Sample Continuation Criteria
• System Test will continue if:- 3. The open bug backlog (“quality gap”)
1. All software released to the Test Team is remains less than 50. The daily and
accompanied by Release Notes rolling closure periods remain less
2. No change is made to the system, than 14 days (on average, bugs are
whether in source code, configuration fixed within two weekly release cycles)
files, or other setup instructions or 4. Twice-weekly bug review meetings
processes, without an accompanying occur until System Test Phase Exit to
bug report. Should a change be made manage the open bug backlog and bug
without a bug report, the Test Manager closure times
will open an urgent bug report
requesting information and escalate to
his manager

Fakulti Teknologi Maklumat dan

Komunikasi
 Exit Criteria

• Continuation criteria measure whether the

test phase can be deemed complete
• Thoroughness measures, such as coverage of
code, functionality or risk
• Estimates of defect density or reliability
measures
• Cost
• Residual risks, such as defects not fixed or lack of
test coverage in certain areas
• Schedules such as those based on time to marker
Fakulti Teknologi Maklumat dan
Komunikasi
 Sample Exit Criteria
• System Test will continue if:- 6. The Test Team has checked that all issues in
1. NO changes (design/code/features), except to the bug tracking system are either closed or
address System Test defects, occurred in the deferred, and where appropriate, verified
prior 3 weeks by regression and confirmation testing
2. NO panic, crash, halt, wedge, unexpected 7. The test metrics indicate product stability
process termination, or other stoppage of and reliability; completion of all planned
processing has occurred on any server tests; adequate coverage of the critical
software or hardware for the previous 3 quality risks
weeks 8. The Project Management Team agrees that
3. No client systems has become inoperable due the product, ad defined during the final
to a failed update during System Test cycle of System Test, will satisfy the
4. The Test Team has executed all the planned customer’s reasonable expectations of
tests against the GA-candidate software quality
5. The Development Teams have resolved all 9. The Project Management Team holds a
“must-fix” bugs per Sales, Marketing and System Test Phase Exit Meeting and agrees
Customer Service that we have completed System Test

Fakulti Teknologi Maklumat dan

Komunikasi
 Developing a Work-Breakdown
Structure (WBS)
• WBS  Is a hierarchical breakdown of the work to
be done on a project
• In each stage, identify the major activities, then
define into discrete tasks
• Ways to identify major activities:-
• “Thinking forward”
• “Thinking backward”
• The tasks should be short in duration with clearly
visible progress markers

Fakulti Teknologi Maklumat dan

Komunikasi
 Estimation
• There are 2 general approaches for estimation:-
1. Estimating individual tasks by the of these tasks or by
experts (bottom up via WBS)
2. Estimating the testing effort based on metrics of
former or similar projects or based on typical values .
• While both are useful, drawing upon team wisdom
to create a WBS, then applying models and rules of
thumb to check and adjust the estimate, is usually
more accurate

Fakulti Teknologi Maklumat dan

Komunikasi
 Factors to consider in Test
Estimation
• Testing is complex, influenced by:
• Process factors: pervasive testing, change control, process maturity,
lifecycle, development and test process, earlier test phases,
estimated vs actual bug levels and fixing
• Material factors: tools, test system, test and debugging
environments, project documentation
• People factors: skills, expectations, suport, relationships
• Delaying factors: complexity, many stakeholders, too much newness,
geographical distribution, need for detailed test documentation
• Understand estimation techniques and these factors
• Deviation from the test estimate can arise from outside factors
and events before or during testing

Fakulti Teknologi Maklumat dan

Komunikasi
 Test Progress Monitoring and
Control
• Common test metrics
• Understand and interpret test metrics
• The purpose and content of the test report
document according to IEEE 829 standard

Fakulti Teknologi Maklumat dan

Komunikasi
 Common Test Metrics
• Percentage completion of test case preparation (or percentage of
planned test cases prepared)
• Percentage completion of preparing the test environment
• Test case execution (e.g no of test cases run/not run, passed/failed)
• Defect information (e.g defect density, defects found and fixed, failure
rate and retest results)
• Coverage of requirements, risks or code by tests, including pass/fail
results
• Level of confidence of testers in the product
• Dates of test milestones
• Testing costs, including the cost of finding the next defect or running
the next test compared to the benefit

Fakulti Teknologi Maklumat dan

Komunikasi
 Test Metrics and Test Reporting
• Test metrics can be used for test reporting
• Test report  is about summarizing or analyzing the test result
• Metric analysis can occur during:
• Key events (e.g meeting, exit criteria)
• Analysis (for recommendation, guidance) of..
• ..cost/benefit of more testing
• ..outstanding risks
• ..level of confidence
• Metrics gathered to assess:-
• Test objectives adequacy for test level
• Test approaches adequacy
• Testing effectiveness per objectives

Fakulti Teknologi Maklumat dan

Komunikasi
 Test Metrics and Test Control
• Guiding and corrective test actions due to
test information and metrics, which may
affect testing activities- or other software life
cycle activities
• Example of test control actions:-
• Risk triggered test re-prioritizing
• Test schedule adjustments due to availability of
test environment
• Set an entry criterion requiring re-testing of bug
fixes by developers before integration into build
Fakulti Teknologi Maklumat dan
Komunikasi
 IEEE 829 Test Summary Report
• A test summary report describes the results if a given level or
phase of testing, and includes the following:-
• Test summary report identifier
• Summary (e.g what was tested, what the conclusions are, etc.)
• Variances (from plan, cases, procedures)
• Comprehensive assessment
• Summary of results (e.g, final metrics, counts)
• Evaluation (of each test item, pass/fail)
• Summary of activities
• Approvals
• May be deliver within or at end of a test level

Fakulti Teknologi Maklumat dan

Komunikasi
• A test log records the relevant details about
test execution, and includes the following
sections:-
• Test logIEEE 829 Test Log
identifier
• Descriptions (item under test[with version
numbers], test environment, etc.)
• Activity and event entries
• Tracking spreadsheets hat capture these
many other details are often used for this
purpose
Fakulti Teknologi Maklumat dan
Komunikasi
 Testing and Configuration Management

• Configuration management establishes, maintains the integrity

of the items that make up the software or system through the
project and product life cycle
• For testing, configuration management:-
• Allows management of testware and results
• Ensures every item in test can be related back to known
system components
• Support delivery of a test release into the test lab
• During project and test planning, the configuration management
procedures and infrastructure (tools) should be chosen,
documented and implemented, so that no surprises occur at
test executing time

Fakulti Teknologi Maklumat dan

Komunikasi
 Key tasks of Configuration Management

• Store and control access to the items that

make up the system
• Identify and document the items under
control
• Allow change to control item through an
orderly process (e.g., change control
boards)
• Report on changes pending, underway, and
complete
• Verify completeness of implementation
Fakulti Teknologi Maklumat dan
Komunikasi
 Test Release Management

• Release schedule (i.e weekly? Daily?

Hourly?)
• Update apply (process to install new build)
• Update unapply (process to remove bad
build)
• Build naming (revision level)
• Interrogation (process to determine rev.
level)
• Synchronizing with database, other systems,
etc Fakulti Teknologi Maklumat dan
Komunikasi
 Risk And Testing

• Testing also subject to risk

• A risk is the possibility of a negative outcome, and that
would include events like late test releases, environment
problems, etc.
• To discover risks to the testing effort, ask yourself or other
stakeholders:
• What would go wrong on the project that would delay or
invalidate your test plan and/or estimate?
• What kind of unacceptable testing outcomes do you worry
about?

Fakulti Teknologi Maklumat dan

Komunikasi
 Handling Project Risks
• For each project risks. You have 4 options:-
• Mitigation : Reduce the likelihood or impact
through preventive steps
• Contingency: Have a plan in place to reduce the
impact
• Transfer: Get some other party to accept the
consequences
• Ignore: Do nothing about it

Fakulti Teknologi Maklumat dan

Komunikasi
 Sample Risks, Mitigations and
Contingencies
• Logistic or product quality problem • Test environment support unreliable
block tests  Good escalation process, system admin skill in
team
 Carefully training, good bug traige,
robust test design  Gaps in test coverage revealed during test
execution
 Test deliverables won’t install  Exploratory testing, continuous test improvement
 Smoke testing, nightly builds,  Slips in test start dates and/or deliverables
uninstall process to test
 Excessive change invalidates results,  Risk priority to drop tests, escalation to
requires test updates management

 Good change control processes,  Budget and/or staffing cuts

robust test design, limited test  Risk priority to drop tests, well rounded team,
outsource testing
documentation, escalation to
management  Debugging in the test environment
 Escalation to management, risk priority to
 Insufficient or unrealistic test drop/reschedule tests
environments
 Explain risks to management,
outsource performance testing

Fakulti Teknologi Maklumat dan

Komunikasi
 Other Project Risks to Consider
• Supplier issues
• Organizational factors • Failure of a third party including a
• Skill and staff shortages test tool vendor
• Contractual issues
• Personal and training issues
• Technical issues
• Communication problems
• Problem in defining the right
• Lack of follow up on test requirements
results, including failure to use • Unachievable requirements
such results for process (leading to blocked/unrunnable
improvements tests)
• The quality of the design, code,
• Improper attitude toward or tests and test data
expectations of testing, • Highly complex system
whether by testers or by others
on the project
• Complex organizations

Fakulti Teknologi Maklumat dan

Komunikasi
 Bug or Incident Management
• Incident or bug reports often have the following goals:
• To provide detailed information about the incident or bug those
who need it.
• To be part of the aggregate data to analyze (e.g bug charts)
• To lead to development and test process improvement
• Typical readers of bug reports include:-
• Developers: who need to fix the problem reported
• Managers: who need to make resource allocation/prioritization
decision about the problem
• Technical support staff: who need to be aware of deferred /unfixed
issues at ship time
• Testers: who need to know the current state of the system

Fakulti Teknologi Maklumat dan

Komunikasi
10 Steps to a Good Bug Report

1. Structure : test carefully

2. Reproduce: test it again
3. Isolate: test it differently
4. Generalize: test it elsewhere
5. Compare: review test similar test results
6. Summarize: relate test to customers
7. Condense: trim unnecessary information
8. Disambiguate: use clear words
9. Neutralize: express problem impartially
10.Review: be sure Fakulti Teknologi Maklumat dan
Komunikasi
Thank You

Fakulti Teknologi Maklumat dan

Komunikasi