SOFTWARE QUALITY Why do programs have bugs?

ASSURANCE
John Abbott College z Involvement makes us blind
z Expectations mask reality

Automated z
z
Interactions are unpredictable
Testing takes too much time
z Testing is repetitive and tedious
Testing
M. E. Kabay, PhD, CISSP
Director of Education, NCSA
President, JINBU Corp

Copyright © 1997 JINBU Corp. All rights reserved

QA 9 - 1 QA 9 - 2

Quality Assurance Depends

The Cost of Software Quality
on Testing
z At least 60% of your development budget z Critical examination
z is used to test only 20-25% z Doing everything feasible to find errors
z of your application features z Errors are deviations from specification

QA 9 - 3 QA 9 - 4

The System Development

To Achieve Software Quality
Life Cycle (SDLC)
Define Requirements
We must conduct a
critical examination Design Specifications
of a system's
quality every time Code Software

we implement a
change Fix Defects

QA 9 - 5 QA 9 - 6
 Why QA? Why automated testing?

z Errors in mission-critical systems can be real- z SDLC known as "waterfall"

world and business disasters z Software backlog running 2 years and more
z E.g., software error in chemical-plant in z JAD/RAD more popular than ever
Netherlands
z Therefore user I/F changes constantly
– allowed operator to enter codes for wrong
chemical substances
– exploded on contact in vat
– destroyed plant
– killed two people

QA 9 - 7 QA 9 - 8

Current Testing Methods are Consequences of Manual

Inadequate Testing Methods
z Manual input z Quality is not emphasized during SDLC
z Unstructured z Time pressures always squeeze testing
z Slow z Testing never catches all the bugs
z Depend on testers' awareness and attention
z Leave no audit trail
z Poor or no statistics
z Manual demonstration of errors

QA 9 - 9 QA 9 - 10

Limitations of
Automated Testing
Capture/Playback
z Capture/Playback z Merely automate manual procedures
– record macros showing mouse z Difficult to maintain as application changes
mouvements and alphanumeric input z Cannot build regression database
– typically no editing language z Must wait until application is ready
z Structured Automated Testing z No mechanism for detecting errors
– tool creates structured, editable script z No mechanism for reporting results
– can use databases as source of input
– intelligent handling of errors

QA 9 - 11 QA 9 - 12
 Good Applications Are Easily
Structured Testing
Maintained and Enhanced
z Structured development z Modular design
z System documentation z Documentation
z Metric: ease of reliably changing application z Segregation of data from procedures
z QA must learn from general programming z Re-usability
experience

QA 9 - 13 QA 9 - 14

Structured Automated Benefits of Structured

Testing Automated Testing
z Define test plan z Consistent, reproducible testing
z Document logic z Increased test coverage
z Generate test procedures z Easier maintenance
z Apply test procedures z Fully documented testing
z Evaluate results z Higher-quality software

QA 9 - 15 QA 9 - 16

Case Study: Case Study:

COGNOS / Ottawa COGNOS / Ottawa
z 15 days for testing z "Our goal was to improve the level of testing
z 6 people @ $300/day while at the same time reducing the time and
manpower for each release."
z 3 test phases per product release
– Doug Clement, COGNOS
z 3000 manual tests per phase
z 12.5% test coverage
z $81,000 per release @ 12.5%
z $648,000 per release @ 100%

QA 9 - 17 QA 9 - 18
 Case Study:
AutoTester at COGNOS
HRL CANADA / Ottawa
z 5 days elapsed time z Jacques Joanisse, manager
z 6 people z Built regression database
z 3 test phases z 80% of the application features were tested
z 24,000 tests/phase z Redeployed 60-80% of test group back into
z $27,000/phase using AutoTester development

QA 9 - 19 QA 9 - 20

Case Study:
Sample ROI
New York Life / Toronto
z Business case showed that up-front cost of z Take $1,000,000 SW development budget
implementing automated QA process will be z Manual testing costs 60% = $600,000
repaid in the first year
z Manual testing: ~20% of application features
z Cost-reduction: lower staffing, shorter cycle
z Automated testing: test 90-95% of features
z Cost-avoidance: identifying more errors
z Automated testing costs 24% = $260,000
z Real savings: $300,000 including cost of
testing tool

QA 9 - 21 QA 9 - 22

Structured Automated PC-based S.A.T. are Platform

Testing Independent
z Emulates trained human operator z Works on PCs
z Single tool for multiple platforms z Can test PC programs
z Supports corporate standards z Can test link to hosts using emulators
z Simplifies training and support z Handles online, real-time systems
z Can deal with batch and hard-copy report
systems by examining results on screen
z Has been used for ATMs, POS, telephone
switches

QA 9 - 23 QA 9 - 24
 Solution to the Software
AutoTester Plus Components
Quality Gap
z Script Station z Structured development is key to effective
– Uses screen image and screen definitions maintenance
– Fully generate all scripts for application z Tie test cases to application specs
tests z Don’t write programs to test
z Test Station application
– Menu-driven application z Test-procedures stored in database
– Uses generated scripts to guide users
– Users create, manage and execute test
cases
– Apply meaningful and comprehensive
tests

QA 9 - 25 QA 9 - 26

Independent Analysis Independent Analysis

Gartner Group:
z "AutoTester from AutoTester, Inc. of Dallas Software Quality Engineering:
reduces testing effort and improves the z "AutoTester is unique because it can test
quality of resulting systems." both PC-based applications as well as
z "AutoTester can effectively provide a facility applications which run on mini or mainframe
for regression testing most interactive computers.
applications at a fraction of the cost and z Variable files support in AutoTester allows
effort required to do so manually." test cases to be imported from external PC or
remote sources, and permits single scripts to
exercise unlimited permutations, thus
conserving script development and
maintenance effort."

QA 9 - 27 QA 9 - 28

Automated Testing is Testing Spans the

Suitable for Many Types of Organization
Quality Assurance Tests
z Way of working
z Unit tests z Not gimmick
z Integration tests z Permeates development team
z Regression tests z Management support required
z Stress tests z Involves users
z Plan for pilot project before choosing tool

QA 9 - 29 QA 9 - 30
 Demonstration Disks Homework
z Readings: Chapters 7, 11, and 13 from your
z Load the demo disk onto a workstation textbook
z Follow the instructions to proceede through z Answer all the review questions distributed
the demonstration. by the instructor
– Avoid copying the textbook blindly -- you
will not remember as much as if you think
about the answers yourself
– Use simple language; usually a few words
or sentences will be ample
z Submit your work by 09:00 tomorrow
morning.
z Because of the short time available, do not be
late in submitting your review answer

QA 9 - 31 QA 9 - 32