CSE 403
System Testing and Test Plans
These lecture slides are copyright (C) Marty Stepp, 2007. They may not be rehosted,
sold, or modified without expressed permission from the author. All rights reserved.
1
Outline
extreme programming
some kinds of system testing
UI testing
build/check-in tests
load testing
usability testing
security testing
monitoring
performance profiling
components of a test plan
2
eXtreme Programming
eXtreme Programming (XP): A software
development approach for small teams on risk-prone
projects with unstable requirements.
created by Kent Beck: Extreme Programming Explained
Testing is a core aspect of Extreme Programming
write unit test code first - before writing the application code
acceptance tests are preferrably automated, and are modified and
rerun for each of the frequent development iterations
QA and test personnel are an integral part of the project team
detailed requirements documentation is not used; frequent re-
scheduling, re-estimating, and re-prioritizing is expected
3
Build tests
build tests (AKA check-in tests)
as code is checked in, these tests (often unit tests) are run
helps find bug regressions
tools
BuildBot: [Link]
SVNAnt: [Link]
Apache Harmony:
[Link]
(some companies roll their own check-in testing systems)
4
UI testing ("acceptance")
automated UI testing ("automation")
scripts and such that use your app and look for failures
a black-box system test
Selenium: [Link]
manual tests
human beings click through predetermined paths
need to write down the specific tests each time
ad-hoc tests
human beings are "turned loose" on the app to see if they can
break it
5
Load testing
load testing
How many hits/requests should the system be able to handle?
What should be its performance under these circumstances?
tools
JMeter: [Link]
curl-loader: [Link]
Database Open-source Test Suite (DOTS):
[Link]
others:
[Link]
6
Usability testing
usability testing: making sure the user can successfully
utilize the system
usability studies
field tests
Some web usability guidelines to consider:
Pages should be 3-5 screens max unless content is tightly
focused on a single topic. If larger, provide internal links.
The page layouts and design elements should be consistent
throughout a site.
Pages should be as browser-independent as possible, or pages
should be provided or generated based on the browser-type.
All pages should have links external to the page; there should
be no dead-end pages.
The page owner, revision date, and a link to a contact person or
organization should be included on each page.
7
Monitoring
monitoring
making sure the site is still up, resources are reachable, etc.
tools
ExternalTest: [Link]
StillUp: [Link]
DotCom-Monitor: [Link]
Internet Vista: [Link]
others:
[Link]
[Link]#MONITORING
8
Web app compatibility
testing
motivation
ensure that your web app is compatible with various browsers,
platforms, etc.
ensure that your app's HTML code complies with web standards
ensure that you have no broken links or other HTML errors
W3C HTML validator: [Link]
W3C link checker: [Link]
9
Testing web applications 1
Questions to ask when testing web apps:
What are the expected loads on the server (e.g., hits/time)?
What kind of performance is required under such loads (server
response time, database query times)?
What kinds of tools will be needed for performance testing (such
as web load testing tools, other tools already in house that can be
adapted, web robot downloading tools, etc.)?
Who is the target audience? What kind of browsers will they be
using? What kind of connection speeds will they by using? Are they
intra- organization (high connection speeds and similar browsers)
or Internet-wide (thus with a wide variety of connection speeds
and browser types)?
What kind of performance is expected on the client side (e.g., how
fast should pages appear, how fast should animations, applets,
etc. load and run)?
Will down time for server and content maintenance/upgrades be
allowed? how much?
What kinds of security (firewalls, encryptions, passwords, etc.) will
be required and what is it expected to do? How can it be tested? 10
Testing web applications 2
How reliable are the site's Internet connections required to be?
And how does that affect backup system or redundant connection
requirements and testing?
What processes will be required to manage updates to the web
site's content, and what are the requirements for maintaining,
tracking, and controlling page content, graphics, links, etc.?
Which HTML specification will be adhered to? How strictly? What
browsers will be allowed?
Will there be any standards or requirements for page appearance
and/or graphics throughout a site or parts of a site?
How will internal and external links be validated and updated? how
often?
Can testing be done on the production system, or will a separate
test system be required? How are browser caching, variations in
browser option settings, dial-up connection variabilities, and real-
world internet 'traffic congestion' problems to be accounted for in
testing?
How extensive or customized are the server logging and reporting
requirements; are they considered an integral part of the system
and do they require testing?
11
Test plans
Outline how your app will be tested in detail
more information:
[Link]
example test plan templates:
[Link]
[Link]
[Link]
example test plan for linux kernel:
[Link]
testing plans and checklists:
[Link]
12
Elements of a test plan 1
Title
Identification of software (incl. version/release #s)
Revision history of document (incl. authors, dates)
Table of Contents
Purpose of document, intended audience
Objective of testing effort
Software product overview
Relevant related document list, such as requirements,
design documents, other test plans, etc.
Relevant standards or legal requirements
Traceability requirements
13
Elements of a test plan 2
Relevant naming conventions and identifier
conventions
Overall software project organization and
personnel/contact-info/responsibilties
Test organization and
personnel/contact-info/responsibilities
Assumptions and dependencies
Project risk analysis
Testing priorities and focus
Scope and limitations of testing
Test outline - a decomposition of the test approach by
test type, feature, functionality, process, system,
module, etc. as applicable
Outline of data input equivalence classes, boundary
value analysis, error classes 14
Elements of a test plan 3
Test environment - hardware, operating systems, other
required software, data configurations, interfaces to
other systems
Test environment validity analysis - differences
between the test and production systems and their
impact on test validity.
Test environment setup and configuration issues
Software migration processes
Software CM processes
Test data setup requirements
Database setup requirements
Outline of system-logging/error-logging/other
capabilities, and tools such as screen capture software,
that will be used to help describe and report bugs
15
Elements of a test plan 4
Discussion of any specialized software or hardware
tools that will be used by testers to help track the cause
or source of bugs
Test automation - justification and overview
Test tools to be used, including versions, patches, etc.
Test script/test code maintenance processes and
version control
Problem tracking and resolution - tools and processes
Project test metrics to be used
Reporting requirements and testing deliverables
Software entrance and exit criteria
Initial sanity testing period and criteria
Test suspension and restart criteria 16
Elements of a test plan 5
Personnel allocation
Personnel pre-training needs
Test site/location
Outside test organizations to be utilized and their
purpose, responsibilties, deliverables, contact persons,
and coordination issues
Relevant proprietary, classified, security, and licensing
issues.
Open issues
Appendix - glossary, acronyms, etc.
17
